[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:23 dist-test-slave-gp6n krb5kdc[28763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:23 dist-test-slave-gp6n krb5kdc[28763](info): set up 2 sockets
Aug 06 02:07:23 dist-test-slave-gp6n krb5kdc[28763](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:07:24.935541 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:25.064234 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46011
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:25.192176 28777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:25.192503 28777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:25.192587 28777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:25.196771 28777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:25.196861 28777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:25.196902 28777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:25.196938 28777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:25.196998 28777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:25.203006 28777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:25.204545 28777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:25.205845 28777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:25.220832 28785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:25.220907 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:25.220850 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:25.221043 28777 server_base.cc:1034] running on GCE node
I20260806 02:07:25.221894 28777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:25.223546 28777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:25.224753 28777 hybrid_clock.cc:648] HybridClock initialized: now 1785982045224707 us; error 58 us; skew 500 ppm
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:25.233821 28777 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:25.235176 28777 webserver.cc:460] Webserver started at http://127.28.21.62:43181/ using document root <none> and password file <none>
I20260806 02:07:25.236024 28777 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:25.236128 28777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:25.236557 28777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:25.239543 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "7751b814b80a4d0cb9768ea9efcbe12e"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "6a63b37a26f66b827348711d18ddb49c"
server_key_iv: "447ed2352011fbab61a39496ae5a630e"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.240217 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "7751b814b80a4d0cb9768ea9efcbe12e"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "6a63b37a26f66b827348711d18ddb49c"
server_key_iv: "447ed2352011fbab61a39496ae5a630e"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.245265 28777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:07:25.248775 28793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.250280 28777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:25.250540 28777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "7751b814b80a4d0cb9768ea9efcbe12e"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "6a63b37a26f66b827348711d18ddb49c"
server_key_iv: "447ed2352011fbab61a39496ae5a630e"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.250662 28777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:25.267760 28777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:25.271839 28777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:25.272101 28777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:25.287750 28777 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:46011
I20260806 02:07:25.287829 28849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:46011 every 8 connection(s)
I20260806 02:07:25.289259 28777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:25.292789 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 28777
I20260806 02:07:25.292937 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:25.294559 28756 external_mini_cluster.cc:1428] Setting key 404999500cdc41a859625b3732f79eb6
I20260806 02:07:25.294930 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:25.305768 28850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: Bootstrap starting.
I20260806 02:07:25.310329 28850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:25.311841 28850 log.cc:824] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:25.314267 28850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: No bootstrap required, opened a new log
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:25.320407 28850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } }
I20260806 02:07:25.320746 28850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:25.320871 28850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7751b814b80a4d0cb9768ea9efcbe12e, State: Initialized, Role: FOLLOWER
I20260806 02:07:25.321913 28850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } }
I20260806 02:07:25.322184 28850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:25.322322 28850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:25.322508 28850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:25.323619 28850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } }
I20260806 02:07:25.324193 28850 leader_election.cc:302] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7751b814b80a4d0cb9768ea9efcbe12e; no voters:
I20260806 02:07:25.324622 28850 leader_election.cc:288] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:25.324779 28855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:25.325066 28855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [term 1 LEADER]: Becoming Leader. State: Replica: 7751b814b80a4d0cb9768ea9efcbe12e, State: Running, Role: LEADER
I20260806 02:07:25.325506 28855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } }
I20260806 02:07:25.326525 28850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:25.328270 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:25.299616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.28.21.62:46011)
0806 02:07:25.300164 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:25.300202 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:25.303643 (+ 3441us) server_negotiation.cc:407] Connection header received
0806 02:07:25.308545 (+ 4902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:25.308562 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:25.308976 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:25.309369 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:25.312378 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.313817 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.315412 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.315987 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.316185 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:25.320961 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:25.321013 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:25.321030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:25.321063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:25.323349 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.323960 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.323965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.323969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.324091 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.324595 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.324601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.324603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.325374 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:25.325622 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:25.326527 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 02:07:25.326928 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":152,"threads_started":1}
I20260806 02:07:25.330754 28856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } } }
I20260806 02:07:25.331048 28856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:25.331929 28857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7751b814b80a4d0cb9768ea9efcbe12e. Latest consensus state: current_term: 1 leader_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7751b814b80a4d0cb9768ea9efcbe12e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46011 } } }
I20260806 02:07:25.332073 28857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e [sys.catalog]: This master's current role is: LEADER
W20260806 02:07:25.335399 28871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:25.335510 28871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:25.335646 28872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:25.336781 28872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:25.344018 28872 catalog_manager.cc:1347] Generated new cluster ID: e96e5460acba4c01a46c92cd5798cde2
I20260806 02:07:25.344099 28872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:25.369582 28872 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:25.371090 28872 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:25.383549 28872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7751b814b80a4d0cb9768ea9efcbe12e: Generated new TSK 0
I20260806 02:07:25.384599 28872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:25.421319 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:25.549602 28877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:25.549880 28877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:25.549938 28877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:25.554023 28877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:25.554085 28877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:25.554144 28877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:25.559727 28877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:25.560772 28877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:25.561781 28877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:25.573016 28882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:25.573092 28883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:25.573107 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:25.573148 28877 server_base.cc:1034] running on GCE node
I20260806 02:07:25.573729 28877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:25.574498 28877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:25.575665 28877 hybrid_clock.cc:648] HybridClock initialized: now 1785982045575645 us; error 29 us; skew 500 ppm
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:25.579228 28877 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:25.580123 28877 webserver.cc:460] Webserver started at http://127.28.21.1:44555/ using document root <none> and password file <none>
I20260806 02:07:25.580662 28877 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:25.580720 28877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:25.580888 28877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:25.583235 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "f07983c2fab04f30befba8b91c52fb66"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "00be977fb8c6a62c7da915103ee4087b"
server_key_iv: "3fb961f2d548c0bc43400b004d405e69"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.583820 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "f07983c2fab04f30befba8b91c52fb66"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "00be977fb8c6a62c7da915103ee4087b"
server_key_iv: "3fb961f2d548c0bc43400b004d405e69"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.587637 28877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 02:07:25.590173 28892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.591449 28877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:07:25.591572 28877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "f07983c2fab04f30befba8b91c52fb66"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "00be977fb8c6a62c7da915103ee4087b"
server_key_iv: "3fb961f2d548c0bc43400b004d405e69"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.591678 28877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:25.605715 28877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:25.609110 28877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:25.609340 28877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:25.610193 28877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:25.611395 28877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:25.611449 28877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.611527 28877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:25.611560 28877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.632432 28877 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:36889
I20260806 02:07:25.632586 29015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:36889 every 8 connection(s)
I20260806 02:07:25.633486 28877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:25.638728 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 28877
I20260806 02:07:25.638882 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:25.639190 28756 external_mini_cluster.cc:1428] Setting key 2a94bd5592ec8c0657833f3a14ce2251
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:25.651084 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:25.635846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:50105 (local address 127.28.21.62:46011)
0806 02:07:25.635998 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:25.636003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:25.637060 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 02:07:25.639134 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:25.639136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:25.639185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:25.639261 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:25.641052 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.641575 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.643833 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.644252 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.644381 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:25.647424 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:25.647427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:25.647429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:25.647452 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:25.648903 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.649448 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.649457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.649458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.649534 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.649884 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.649887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.649889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.650077 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:25.650182 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:25.650763 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:07:25.650891 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:25.651954 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:25.636290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46011 (local address 127.28.21.1:50105)
0806 02:07:25.636878 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:25.636927 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:25.638887 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:25.639573 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:25.639582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:25.640145 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:25.640883 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:25.640906 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.641738 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.641744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:25.643699 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:25.643709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.644440 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.644443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:25.644587 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:25.645271 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:25.645296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:25.647249 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:25.649026 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:25.649033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:25.649048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:25.649333 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:25.649627 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:25.649631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:25.649633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:25.649778 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:25.650185 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:25.650194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:25.650509 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 02:07:25.650704 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:07:25.651035 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":171,"threads_started":1}
I20260806 02:07:25.653371 29016 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46011
I20260806 02:07:25.653820 29016 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:25.654306 29016 heartbeater.cc:511] Master 127.28.21.62:46011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:25.656553 28812 ts_manager.cc:194] Registered new tserver with Master: f07983c2fab04f30befba8b91c52fb66 (127.28.21.1:36889)
I20260806 02:07:25.659168 28812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:50105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:25.674341 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:25.814447 29021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:25.814777 29021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:25.814889 29021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:25.819069 29021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:25.819152 29021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:25.819242 29021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:25.824884 29021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:25.826018 29021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:25.827167 29021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:25.837867 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:25.837865 29026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:25.838203 29021 server_base.cc:1034] running on GCE node
W20260806 02:07:25.838052 29027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:25.838778 29021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:25.839383 29021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:25.840569 29021 hybrid_clock.cc:648] HybridClock initialized: now 1785982045840534 us; error 44 us; skew 500 ppm
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:25.844084 29021 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:25.845114 29021 webserver.cc:460] Webserver started at http://127.28.21.2:41721/ using document root <none> and password file <none>
I20260806 02:07:25.845778 29021 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:25.845880 29021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:25.846144 29021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:25.848497 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "27249de857be4d71973fd75f1708bfa4"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "9022bf126ed4a64a2daf28cca3152644"
server_key_iv: "2889b6d8de371294f4a94290a6e9e9b9"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.849123 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "27249de857be4d71973fd75f1708bfa4"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "9022bf126ed4a64a2daf28cca3152644"
server_key_iv: "2889b6d8de371294f4a94290a6e9e9b9"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.853344 29021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:07:25.856151 29036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.857462 29021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:07:25.857621 29021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "27249de857be4d71973fd75f1708bfa4"
format_stamp: "Formatted at 2026-08-06 02:07:25 on dist-test-slave-gp6n"
server_key: "9022bf126ed4a64a2daf28cca3152644"
server_key_iv: "2889b6d8de371294f4a94290a6e9e9b9"
server_key_version: "encryptionkey@0"
I20260806 02:07:25.857748 29021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:25.887831 29021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:25.891459 29021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:25.891682 29021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:25.892320 29021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:25.893178 29021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:25.893249 29021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.893323 29021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:25.893368 29021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:25.903231 29021 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:33399
I20260806 02:07:25.903326 29157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:33399 every 8 connection(s)
I20260806 02:07:25.904181 29021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:25.912158 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29021
I20260806 02:07:25.912285 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:25.912593 28756 external_mini_cluster.cc:1428] Setting key ba08953844fe8c60078502e6893f0c6e
Aug 06 02:07:25 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:25.919373 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:25.906174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:43683 (local address 127.28.21.62:46011)
0806 02:07:25.906332 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:25.906337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:25.907256 (+ 919us) server_negotiation.cc:407] Connection header received
0806 02:07:25.908539 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:25.908542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:25.908589 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:25.908672 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:25.909853 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.910394 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.911600 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.911830 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.911980 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:25.914941 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:25.914944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:25.914946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:25.914970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:25.916892 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.917419 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.917423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.917425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.917523 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:25.917996 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:25.918001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:25.918003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:25.918212 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:25.918310 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:25.919074 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 02:07:25.919199 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:07:25.920629 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:25.906642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46011 (local address 127.28.21.2:43683)
0806 02:07:25.907100 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:25.907145 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:25.908324 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:25.908844 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:25.908851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:25.909210 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:25.909698 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:25.909711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.910594 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.910597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:25.911485 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:25.911494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:25.912007 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:25.912010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:25.912155 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:25.912794 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:25.912816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:25.914729 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:25.917021 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:25.917026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:25.917039 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:25.917308 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:25.917642 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:25.917647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:25.917649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:25.917855 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:25.918378 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:25.918388 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:25.918769 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 02:07:25.919015 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:07:25.919447 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260806 02:07:25.922024 29158 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46011
I20260806 02:07:25.922359 29158 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:25.922919 29158 heartbeater.cc:511] Master 127.28.21.62:46011 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:25.924098 28812 ts_manager.cc:194] Registered new tserver with Master: 27249de857be4d71973fd75f1708bfa4 (127.28.21.2:33399)
I20260806 02:07:25.924721 28812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:43683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:25.943547 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:26.060624 29164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:26.060916 29164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:26.060963 29164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:26.065282 29164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:26.065412 29164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:26.065477 29164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:26.071015 29164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:26.072152 29164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:26.073225 29164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:26.083483 29169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:26.083492 29172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:26.083588 29164 server_base.cc:1034] running on GCE node
W20260806 02:07:26.083549 29170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:26.084139 29164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:26.084883 29164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:26.086097 29164 hybrid_clock.cc:648] HybridClock initialized: now 1785982046086060 us; error 48 us; skew 500 ppm
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:26.089149 29164 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:26.090183 29164 webserver.cc:460] Webserver started at http://127.28.21.3:45623/ using document root <none> and password file <none>
I20260806 02:07:26.090802 29164 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:26.090869 29164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:26.091039 29164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:26.093122 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "912bd234a2dc41958f67a730f7d66b6f"
format_stamp: "Formatted at 2026-08-06 02:07:26 on dist-test-slave-gp6n"
server_key: "83aeffa462dccdbac8a4db107fa06854"
server_key_iv: "22d9abf82d7ca801e2c0bd91268c4e10"
server_key_version: "encryptionkey@0"
I20260806 02:07:26.093642 29164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "912bd234a2dc41958f67a730f7d66b6f"
format_stamp: "Formatted at 2026-08-06 02:07:26 on dist-test-slave-gp6n"
server_key: "83aeffa462dccdbac8a4db107fa06854"
server_key_iv: "22d9abf82d7ca801e2c0bd91268c4e10"
server_key_version: "encryptionkey@0"
I20260806 02:07:26.097541 29164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:07:26.100226 29179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:26.101475 29164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:26.101581 29164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "912bd234a2dc41958f67a730f7d66b6f"
format_stamp: "Formatted at 2026-08-06 02:07:26 on dist-test-slave-gp6n"
server_key: "83aeffa462dccdbac8a4db107fa06854"
server_key_iv: "22d9abf82d7ca801e2c0bd91268c4e10"
server_key_version: "encryptionkey@0"
I20260806 02:07:26.101670 29164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:26.125262 29164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:26.128716 29164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:26.128888 29164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:26.129515 29164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:26.130357 29164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:26.130462 29164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:26.130527 29164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:26.130566 29164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:26.139936 29164 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:42645
I20260806 02:07:26.139968 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:42645 every 8 connection(s)
I20260806 02:07:26.140902 29164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:26.151060 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29164
I20260806 02:07:26.151213 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:26.151489 28756 external_mini_cluster.cc:1428] Setting key a984d58e48f6e790e28ef13a558a427e
I20260806 02:07:26.156129 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:26.142852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59877 (local address 127.28.21.62:46011)
0806 02:07:26.143035 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:26.143039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:26.143785 (+ 746us) server_negotiation.cc:407] Connection header received
0806 02:07:26.145203 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:26.145207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:26.145272 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:26.145425 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:26.146739 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.147340 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.148605 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.148853 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.148972 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:26.151775 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:26.151779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:26.151781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:26.151813 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:26.153824 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.154523 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.154527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.154530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.154629 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.155071 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.155074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.155075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.155299 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:26.155377 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:26.155901 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 02:07:26.156004 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:07:26.157084 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:26.143134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46011 (local address 127.28.21.3:59877)
0806 02:07:26.143616 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:26.143662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:26.144968 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:26.145622 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:26.145629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:26.146034 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:26.146574 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:26.146590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.147483 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.147488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:26.148464 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:26.148473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.149027 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.149032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:26.149221 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:26.149865 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:26.149882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:26.151517 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:26.153953 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:26.153959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:26.153975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:26.154362 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:26.154748 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:26.154751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:26.154753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:26.154958 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:26.155404 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:26.155409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:26.155669 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:07:26.155823 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 02:07:26.156154 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260806 02:07:26.158326 29301 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46011
I20260806 02:07:26.158629 29301 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:26.159137 29301 heartbeater.cc:511] Master 127.28.21.62:46011 requested a full tablet report, sending...
I20260806 02:07:26.160212 28812 ts_manager.cc:194] Registered new tserver with Master: 912bd234a2dc41958f67a730f7d66b6f (127.28.21.3:42645)
I20260806 02:07:26.160799 28812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:59877
I20260806 02:07:26.166568 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:26.196875 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:26.186516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.28.21.1:36889)
0806 02:07:26.186907 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:26.186913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:26.186943 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:07:26.187087 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:26.187093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:26.187294 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:26.187465 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:26.188154 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.188994 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.189931 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.190315 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.190469 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:26.193005 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:26.193016 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:26.193030 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:26.193055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:26.194908 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.195317 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.195322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.195324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.195488 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.195759 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.195761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.195762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.196075 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:26.196237 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:26.196498 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:07:26.196700 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
W20260806 02:07:26.197873 28921 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40554
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[28763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:26.228080 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:26.216106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.28.21.1:36889)
0806 02:07:26.216283 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:26.216292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:26.216470 (+ 178us) server_negotiation.cc:407] Connection header received
0806 02:07:26.216630 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:26.216640 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:26.216689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:26.216804 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:26.217618 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.218524 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.219881 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:26.220257 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:26.220336 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:26.223744 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:26.223749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:26.223752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:26.223784 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:26.225702 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.226262 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.226267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.226269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.226381 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:26.226815 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:26.226820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:26.226826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:26.227212 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:26.227379 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:26.227717 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:07:26.227879 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:07:26.229544 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 28877
I20260806 02:07:26.238250 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29021
I20260806 02:07:26.246371 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29164
I20260806 02:07:26.252720 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 28777
2026-08-06T02:07:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2407 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[29317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[29317](info): set up 2 sockets
Aug 06 02:07:26 dist-test-slave-gp6n krb5kdc[29317](info): commencing operation
krb5kdc: starting...
W20260806 02:07:27.348996 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:27 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:27.475476 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38455
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:38455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:27.586083 29332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:27.586357 29332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:27.586458 29332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:27.590368 29332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:27.590440 29332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:27.590461 29332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:27.590477 29332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:27.590493 29332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:27.595665 29332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:38455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:27.596627 29332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:27.597586 29332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:27.606511 29338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:27.606482 29341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:27.606541 29339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:27.606846 29332 server_base.cc:1034] running on GCE node
I20260806 02:07:27.607583 29332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:27.608739 29332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:27.609948 29332 hybrid_clock.cc:648] HybridClock initialized: now 1785982047609931 us; error 27 us; skew 500 ppm
Aug 06 02:07:27 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:27.612831 29332 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:27.613745 29332 webserver.cc:460] Webserver started at http://127.28.21.62:40171/ using document root <none> and password file <none>
I20260806 02:07:27.614275 29332 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:27.614333 29332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:27.614573 29332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:27.616698 29332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "3560c1c5f52f4bf99c727662593c4dba"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "26ee1eab0cc78fbe623ad25631a06a36"
server_key_iv: "83b0a78f127588a3e9d2fe74d7629b6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.617230 29332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "3560c1c5f52f4bf99c727662593c4dba"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "26ee1eab0cc78fbe623ad25631a06a36"
server_key_iv: "83b0a78f127588a3e9d2fe74d7629b6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.620926 29332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:07:27.623469 29349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:27.624687 29332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:27.624789 29332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "3560c1c5f52f4bf99c727662593c4dba"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "26ee1eab0cc78fbe623ad25631a06a36"
server_key_iv: "83b0a78f127588a3e9d2fe74d7629b6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.624876 29332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:27.647637 29332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:27.650564 29332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:27.650729 29332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:27.658475 29332 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:38455
I20260806 02:07:27.658494 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:38455 every 8 connection(s)
I20260806 02:07:27.659471 29332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:27.661849 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29332
I20260806 02:07:27.661947 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:27.662220 28756 external_mini_cluster.cc:1428] Setting key 0cc4348126eda5944810f87c1b8a401c
I20260806 02:07:27.662370 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:27.668905 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba: Bootstrap starting.
Aug 06 02:07:27 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:27.671841 29408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:27.672597 29408 log.cc:824] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:27.675047 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba: No bootstrap required, opened a new log
I20260806 02:07:27.677716 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:27.663734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.28.21.62:38455)
0806 02:07:27.664184 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:27.664198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:27.664233 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:07:27.665211 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:27.665230 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:27.665553 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:27.665891 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:27.666728 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.667732 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.668588 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.669105 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.669251 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:27.672221 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:27.672227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:27.672239 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:27.672267 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:27.674360 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:27.674843 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:27.674849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:27.674854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:27.674997 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:27.675423 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:27.675429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:27.675430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:27.675933 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:27.676136 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:27.676336 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 02:07:27.676644 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":150,"threads_started":1}
I20260806 02:07:27.679718 29408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } }
I20260806 02:07:27.679996 29408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:27.680095 29408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3560c1c5f52f4bf99c727662593c4dba, State: Initialized, Role: FOLLOWER
I20260806 02:07:27.680486 29408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } }
I20260806 02:07:27.680627 29408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:27.680689 29408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:27.680809 29408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:27.681706 29408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } }
I20260806 02:07:27.682049 29408 leader_election.cc:302] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3560c1c5f52f4bf99c727662593c4dba; no voters:
I20260806 02:07:27.682345 29408 leader_election.cc:288] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:27.682509 29413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:27.682756 29413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [term 1 LEADER]: Becoming Leader. State: Replica: 3560c1c5f52f4bf99c727662593c4dba, State: Running, Role: LEADER
I20260806 02:07:27.683125 29413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } }
I20260806 02:07:27.683573 29408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:27.684670 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3560c1c5f52f4bf99c727662593c4dba. Latest consensus state: current_term: 1 leader_uuid: "3560c1c5f52f4bf99c727662593c4dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } } }
I20260806 02:07:27.684793 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:27.685184 29422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:27.686007 29414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3560c1c5f52f4bf99c727662593c4dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3560c1c5f52f4bf99c727662593c4dba" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38455 } } }
I20260806 02:07:27.686184 29414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:27.688280 29422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:27.693900 29422 catalog_manager.cc:1347] Generated new cluster ID: 26dd3a633dfa4be88cbb669a349e6ed5
I20260806 02:07:27.693987 29422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:27.712630 29422 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:27.713912 29422 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:27.734138 29422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3560c1c5f52f4bf99c727662593c4dba: Generated new TSK 0
I20260806 02:07:27.734946 29422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:27.769635 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:27.882076 29435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:27.882395 29435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:27.882526 29435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:27.886935 29435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:27.887001 29435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:27.887064 29435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:27.892421 29435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:27.893467 29435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:27.894536 29435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:27.904619 29442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:27.904575 29441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:27.904724 29435 server_base.cc:1034] running on GCE node
W20260806 02:07:27.904634 29444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:27.905277 29435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:27.905977 29435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:27.907158 29435 hybrid_clock.cc:648] HybridClock initialized: now 1785982047907143 us; error 29 us; skew 500 ppm
Aug 06 02:07:27 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:27.910092 29435 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:27.911049 29435 webserver.cc:460] Webserver started at http://127.28.21.1:38537/ using document root <none> and password file <none>
I20260806 02:07:27.911615 29435 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:27.911679 29435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:27.911913 29435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:27.913949 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "56b7cfbe81cd4615adbaf61d7b046b88"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "ce1377dae0bd968ce11bfe7196e266d1"
server_key_iv: "b131692251f92a11c30a28365a382353"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.914587 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "56b7cfbe81cd4615adbaf61d7b046b88"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "ce1377dae0bd968ce11bfe7196e266d1"
server_key_iv: "b131692251f92a11c30a28365a382353"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.918229 29435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:27.920703 29451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:27.921949 29435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:27.922088 29435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "56b7cfbe81cd4615adbaf61d7b046b88"
format_stamp: "Formatted at 2026-08-06 02:07:27 on dist-test-slave-gp6n"
server_key: "ce1377dae0bd968ce11bfe7196e266d1"
server_key_iv: "b131692251f92a11c30a28365a382353"
server_key_version: "encryptionkey@0"
I20260806 02:07:27.922201 29435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:27.936342 29435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:27.939235 29435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:27.939443 29435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:27.940035 29435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:27.940862 29435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:27.940935 29435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:27.941004 29435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:27.941038 29435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:27.950105 29435 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:42837
I20260806 02:07:27.950129 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:42837 every 8 connection(s)
I20260806 02:07:27.951061 29435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:27.956854 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29435
I20260806 02:07:27.956984 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:27.957269 28756 external_mini_cluster.cc:1428] Setting key e4395df0ca97bca6cb31d45bbcc84cfb
Aug 06 02:07:27 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:27.965668 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:27.952879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:54231 (local address 127.28.21.62:38455)
0806 02:07:27.953017 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:27.953021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:27.953826 (+ 805us) server_negotiation.cc:407] Connection header received
0806 02:07:27.954736 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:27.954739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:27.954787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:27.954863 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:27.956042 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.956583 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.958065 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.958311 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.958449 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:27.961424 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:27.961427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:27.961429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:27.961456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:27.963337 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:27.963870 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:27.963874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:27.963876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:27.963984 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:27.964336 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:27.964342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:27.964347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:27.964616 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:27.964731 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:27.965248 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 02:07:27.965426 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:27.966320 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:27.953211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38455 (local address 127.28.21.1:54231)
0806 02:07:27.953662 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:27.953704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:27.954541 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:27.955026 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:27.955032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:27.955375 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:27.955898 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:27.955911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.956710 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.956713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:27.957928 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:27.957937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:27.958493 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:27.958497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:27.958706 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:27.959295 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:27.959319 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:27.961232 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:27.963467 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:27.963472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:27.963486 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:27.963758 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:27.964090 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:27.964092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:27.964094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:27.964237 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:27.964742 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:27.964747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:27.964994 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 02:07:27.965163 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:07:27.965417 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":104,"threads_started":1}
I20260806 02:07:27.967732 29574 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38455
I20260806 02:07:27.967980 29574 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:27.968531 29574 heartbeater.cc:511] Master 127.28.21.62:38455 requested a full tablet report, sending...
I20260806 02:07:27.970248 29368 ts_manager.cc:194] Registered new tserver with Master: 56b7cfbe81cd4615adbaf61d7b046b88 (127.28.21.1:42837)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:27.971779 29368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:54231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:27.987224 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:28.101688 29580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:28.101960 29580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:28.102006 29580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:28.106503 29580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:28.106563 29580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:28.106673 29580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:28.112411 29580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:28.113564 29580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:28.114701 29580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:28.125078 29588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:28.125079 29585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:28.125196 29580 server_base.cc:1034] running on GCE node
W20260806 02:07:28.125139 29586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:28.125839 29580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:28.126646 29580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:28.127875 29580 hybrid_clock.cc:648] HybridClock initialized: now 1785982048127845 us; error 43 us; skew 500 ppm
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:28.130811 29580 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:28.131744 29580 webserver.cc:460] Webserver started at http://127.28.21.2:37739/ using document root <none> and password file <none>
I20260806 02:07:28.132329 29580 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:28.132424 29580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:28.132668 29580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:28.134827 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "c12a5e0e0764494aaa0e08a6285deeb8"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "1ce09871e82288105e7a76693a3b6ba0"
server_key_iv: "62673b0d5a84797b85105f7176656401"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.135421 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "c12a5e0e0764494aaa0e08a6285deeb8"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "1ce09871e82288105e7a76693a3b6ba0"
server_key_iv: "62673b0d5a84797b85105f7176656401"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.139356 29580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:07:28.141875 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.143143 29580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:28.143288 29580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "c12a5e0e0764494aaa0e08a6285deeb8"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "1ce09871e82288105e7a76693a3b6ba0"
server_key_iv: "62673b0d5a84797b85105f7176656401"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.143406 29580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:28.152386 29580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:28.155776 29580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:28.155997 29580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:28.156642 29580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:28.157508 29580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:28.157591 29580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.157666 29580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:28.157709 29580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.167286 29580 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:42931
I20260806 02:07:28.167312 29715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:42931 every 8 connection(s)
I20260806 02:07:28.168241 29580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:28.173888 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29580
I20260806 02:07:28.173985 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:28.174283 28756 external_mini_cluster.cc:1428] Setting key 36cab25bc208a23a74505c431011418a
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:28.182803 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.170087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:56997 (local address 127.28.21.62:38455)
0806 02:07:28.170226 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.170229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.171104 (+ 875us) server_negotiation.cc:407] Connection header received
0806 02:07:28.172023 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.172026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.172088 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.172192 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.173387 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.174095 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.175339 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.175579 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.175692 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.178756 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.178777 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.178782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.178813 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.180585 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.181106 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.181110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.181112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.181201 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.181553 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.181556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.181558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.181750 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.181865 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.182566 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.182680 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:28.184098 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.170468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38455 (local address 127.28.21.2:56997)
0806 02:07:28.170952 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.170996 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.171798 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.172371 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.172380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.172744 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:28.173227 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.173240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.174225 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.174233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.175207 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.175216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.175730 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.175733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.175907 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.176554 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:28.176582 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:28.178485 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:28.180707 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:28.180714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:28.180729 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:28.180995 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:28.181293 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:28.181297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:28.181299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:28.181447 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:28.181859 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:28.181865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:28.182205 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.182474 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.182862 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260806 02:07:28.185655 29716 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38455
I20260806 02:07:28.185918 29716 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:28.186543 29716 heartbeater.cc:511] Master 127.28.21.62:38455 requested a full tablet report, sending...
I20260806 02:07:28.187619 29368 ts_manager.cc:194] Registered new tserver with Master: c12a5e0e0764494aaa0e08a6285deeb8 (127.28.21.2:42931)
I20260806 02:07:28.188212 29368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:56997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:28.204919 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:28.318455 29721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:28.318737 29721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:28.318815 29721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:28.322870 29721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:28.322944 29721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:28.323032 29721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:28.328428 29721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:28.329516 29721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:28.330623 29721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:28.340296 29727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:28.340312 29729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:28.340350 29721 server_base.cc:1034] running on GCE node
W20260806 02:07:28.340310 29726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:28.340909 29721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:28.341516 29721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:28.342697 29721 hybrid_clock.cc:648] HybridClock initialized: now 1785982048342669 us; error 44 us; skew 500 ppm
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:28.345302 29721 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:28.346170 29721 webserver.cc:460] Webserver started at http://127.28.21.3:32847/ using document root <none> and password file <none>
I20260806 02:07:28.346741 29721 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:28.346804 29721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:28.346972 29721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:28.349002 29721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "f21df0b6b1104f3fac61b045290acd20"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "4012483e105cff43947ae75667a3e636"
server_key_iv: "305a9f5803326700deadd7ce9b3fe3e5"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.349526 29721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "f21df0b6b1104f3fac61b045290acd20"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "4012483e105cff43947ae75667a3e636"
server_key_iv: "305a9f5803326700deadd7ce9b3fe3e5"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.353202 29721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260806 02:07:28.355674 29737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.356910 29721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:28.357056 29721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "f21df0b6b1104f3fac61b045290acd20"
format_stamp: "Formatted at 2026-08-06 02:07:28 on dist-test-slave-gp6n"
server_key: "4012483e105cff43947ae75667a3e636"
server_key_iv: "305a9f5803326700deadd7ce9b3fe3e5"
server_key_version: "encryptionkey@0"
I20260806 02:07:28.357170 29721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:28.383096 29721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:28.386250 29721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:28.386487 29721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:28.387096 29721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:28.387924 29721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:28.387979 29721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.388047 29721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:28.388087 29721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:28.397176 29721 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:45133
I20260806 02:07:28.397207 29861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:45133 every 8 connection(s)
I20260806 02:07:28.398105 29721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:28.401068 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29721
I20260806 02:07:28.401196 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:28.401476 28756 external_mini_cluster.cc:1428] Setting key 6a3862143a76d569be50cd7c4d89cc1c
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:28.411262 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.399885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:42987 (local address 127.28.21.62:38455)
0806 02:07:28.400039 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.400043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.400835 (+ 792us) server_negotiation.cc:407] Connection header received
0806 02:07:28.402053 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.402057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.402105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.402211 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.403581 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.404272 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.405395 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.405615 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.405696 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.408201 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.408204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.408206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.408228 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.409460 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.409924 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.409926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.409927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.410009 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.410297 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.410299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.410300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.410523 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.410601 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.411048 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.411144 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:07:28.412077 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.400242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38455 (local address 127.28.21.3:42987)
0806 02:07:28.400673 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.400715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.401782 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.402441 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.402453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.402868 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:28.403424 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.403462 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.404391 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.404394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.405284 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.405293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.405743 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.405745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.405883 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.406536 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:28.406552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:28.408053 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:28.409566 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:28.409570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:28.409582 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:28.409839 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:28.410088 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:28.410090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:28.410091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:28.410219 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:28.410612 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:28.410617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:28.410838 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.410993 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.411250 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260806 02:07:28.413204 29862 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38455
I20260806 02:07:28.413441 29862 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:28.413911 29862 heartbeater.cc:511] Master 127.28.21.62:38455 requested a full tablet report, sending...
I20260806 02:07:28.415002 29368 ts_manager.cc:194] Registered new tserver with Master: f21df0b6b1104f3fac61b045290acd20 (127.28.21.3:45133)
I20260806 02:07:28.415556 29368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:42987
I20260806 02:07:28.415766 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:07:28.426954 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.419922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.28.21.62:38455)
0806 02:07:28.420071 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.420077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.420267 (+ 190us) server_negotiation.cc:407] Connection header received
0806 02:07:28.420383 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.420386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.420423 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.420487 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.421150 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.421630 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.422395 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.422649 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.422726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.423928 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.423931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.423933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.423953 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.425359 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.425715 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.425717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.425718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.425790 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.426090 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.426092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.426093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.426290 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.426392 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.426727 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.426818 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:28.432474 29368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:28.435786 29368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:28.450569 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.444816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:42931 (local address 127.0.0.1:33166)
0806 02:07:28.445284 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.445327 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.445450 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.446250 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.446256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.446279 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:28.446429 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.446438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.447843 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.447848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.448883 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.448893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.450005 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.450132 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.450169 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.450280 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.450401 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":85,"threads_started":1}
I20260806 02:07:28.450575 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.444953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.28.21.2:42931)
0806 02:07:28.445376 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.445390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.445416 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:07:28.445660 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.445667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.445869 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.446046 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:28.446562 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.447707 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.449026 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.449877 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450005 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.450210 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.450313 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.450425 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":112,"threads_started":1}
I20260806 02:07:28.451354 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.444893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:45133 (local address 127.0.0.1:37474)
0806 02:07:28.445767 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.445780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.445875 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.447028 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.447031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.447050 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:28.447155 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.447160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.448575 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.448578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.449669 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.449678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.450922 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.451041 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.451071 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.451151 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.451221 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":5,"thread_start_us":125,"threads_started":1}
I20260806 02:07:28.451560 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.445616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.28.21.3:45133)
0806 02:07:28.446397 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.446430 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.446462 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:07:28.446538 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.446545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.446732 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.446876 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:28.447278 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.448421 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.449797 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.450668 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450776 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.450908 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.451209 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.451301 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":152,"threads_started":1}
I20260806 02:07:28.451666 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.446053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.28.21.1:42837)
0806 02:07:28.446364 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.446380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.446423 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:07:28.446500 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.446504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.446702 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.446858 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:28.447191 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.448385 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.449871 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.450668 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450776 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.450908 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.451419 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.451515 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":140,"threads_started":1}
I20260806 02:07:28.451887 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.445514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42837 (local address 127.0.0.1:44856)
0806 02:07:28.446013 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.446027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.446104 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.446895 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.446900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.446922 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:28.447038 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.447045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.448545 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.448548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.449757 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.449768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.450981 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.450984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.451117 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.451141 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.451217 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.451289 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":32,"thread_start_us":95,"threads_started":1}
I20260806 02:07:28.454104 29788 tablet_service.cc:1467] Processing CreateTablet for tablet f4d6be7271b44db6870bafb6eed32dd3 (DEFAULT_TABLE table=test-table [id=ee6d7babbe724b389ab07c205055dc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:28.454100 29648 tablet_service.cc:1467] Processing CreateTablet for tablet f4d6be7271b44db6870bafb6eed32dd3 (DEFAULT_TABLE table=test-table [id=ee6d7babbe724b389ab07c205055dc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:28.455377 29788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d6be7271b44db6870bafb6eed32dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:28.456437 29507 tablet_service.cc:1467] Processing CreateTablet for tablet f4d6be7271b44db6870bafb6eed32dd3 (DEFAULT_TABLE table=test-table [id=ee6d7babbe724b389ab07c205055dc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:28.457350 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d6be7271b44db6870bafb6eed32dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:28.457401 29507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d6be7271b44db6870bafb6eed32dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:28.465881 29881 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: Bootstrap starting.
I20260806 02:07:28.467959 29881 tablet_bootstrap.cc:654] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:28.468684 29881 log.cc:824] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:28.469390 29880 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: Bootstrap starting.
I20260806 02:07:28.470815 29879 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: Bootstrap starting.
I20260806 02:07:28.472026 29880 tablet_bootstrap.cc:654] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:28.472093 29881 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: No bootstrap required, opened a new log
I20260806 02:07:28.472275 29881 ts_tablet_manager.cc:1397] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:07:28.472710 29879 tablet_bootstrap.cc:654] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:28.472906 29880 log.cc:824] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:28.474035 29879 log.cc:824] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:28.475102 29880 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: No bootstrap required, opened a new log
I20260806 02:07:28.475291 29880 ts_tablet_manager.cc:1397] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:07:28.475927 29879 tablet_bootstrap.cc:492] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: No bootstrap required, opened a new log
I20260806 02:07:28.476125 29879 ts_tablet_manager.cc:1397] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:28.476383 29881 raft_consensus.cc:348] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.476663 29881 raft_consensus.cc:374] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:28.476718 29881 raft_consensus.cc:729] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b7cfbe81cd4615adbaf61d7b046b88, State: Initialized, Role: FOLLOWER
I20260806 02:07:28.477151 29881 consensus_queue.cc:260] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.478132 29881 ts_tablet_manager.cc:1428] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:07:28.478616 29574 heartbeater.cc:503] Master 127.28.21.62:38455 was elected leader, sending a full tablet report...
I20260806 02:07:28.479177 29880 raft_consensus.cc:348] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.479413 29880 raft_consensus.cc:374] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:28.479471 29880 raft_consensus.cc:729] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12a5e0e0764494aaa0e08a6285deeb8, State: Initialized, Role: FOLLOWER
I20260806 02:07:28.479919 29880 consensus_queue.cc:260] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.480230 29879 raft_consensus.cc:348] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.480553 29879 raft_consensus.cc:374] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:28.480615 29879 raft_consensus.cc:729] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21df0b6b1104f3fac61b045290acd20, State: Initialized, Role: FOLLOWER
I20260806 02:07:28.480711 29716 heartbeater.cc:503] Master 127.28.21.62:38455 was elected leader, sending a full tablet report...
I20260806 02:07:28.480960 29880 ts_tablet_manager.cc:1428] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:07:28.481065 29879 consensus_queue.cc:260] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.481734 29862 heartbeater.cc:503] Master 127.28.21.62:38455 was elected leader, sending a full tablet report...
I20260806 02:07:28.482055 29879 ts_tablet_manager.cc:1428] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:07:28.512430 29885 raft_consensus.cc:482] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:28.512669 29885 raft_consensus.cc:504] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.513863 29885 leader_election.cc:288] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12a5e0e0764494aaa0e08a6285deeb8 (127.28.21.2:42931), f21df0b6b1104f3fac61b045290acd20 (127.28.21.3:45133)
I20260806 02:07:28.517944 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.514217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:57481 (local address 127.28.21.2:42931)
0806 02:07:28.514447 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.514453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.514562 (+ 109us) server_negotiation.cc:407] Connection header received
0806 02:07:28.514681 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.514686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.514741 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.514819 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:28.515151 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.515695 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.516877 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.517307 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.517392 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.517500 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.517727 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.517816 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:07:28.517967 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.514079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:42931 (local address 127.28.21.1:57481)
0806 02:07:28.514445 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.514459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.514576 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.514843 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.514845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.514862 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:28.515022 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.515030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.515809 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.515815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.516761 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.516770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.517451 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.517455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.517603 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.517622 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.517708 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.517798 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":4,"thread_start_us":132,"threads_started":1}
I20260806 02:07:28.517985 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.514126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:45133 (local address 127.28.21.1:38943)
0806 02:07:28.514623 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:28.514638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:28.514739 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:28.515004 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:28.515006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:28.515020 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:28.515149 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.515156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.515875 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.515878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.516805 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:28.516831 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.517481 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.517485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:28.517693 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.517720 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:07:28.517791 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:07:28.517854 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":75,"thread_start_us":117,"threads_started":1}
I20260806 02:07:28.518020 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.514247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:38943 (local address 127.28.21.3:45133)
0806 02:07:28.514449 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.514455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.514726 (+ 271us) server_negotiation.cc:407] Connection header received
0806 02:07:28.514851 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.514854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.514905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.514983 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:28.515257 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.515735 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.516941 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.517328 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.517450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.517513 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.517822 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.517898 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 02:07:28.518720 29810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d6be7271b44db6870bafb6eed32dd3" candidate_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21df0b6b1104f3fac61b045290acd20" is_pre_election: true
I20260806 02:07:28.518726 29669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d6be7271b44db6870bafb6eed32dd3" candidate_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" is_pre_election: true
I20260806 02:07:28.519132 29810 raft_consensus.cc:2454] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b7cfbe81cd4615adbaf61d7b046b88 in term 0.
I20260806 02:07:28.519141 29669 raft_consensus.cc:2454] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b7cfbe81cd4615adbaf61d7b046b88 in term 0.
I20260806 02:07:28.519683 29454 leader_election.cc:302] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b7cfbe81cd4615adbaf61d7b046b88, f21df0b6b1104f3fac61b045290acd20; no voters:
I20260806 02:07:28.519985 29885 raft_consensus.cc:2781] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:28.520074 29885 raft_consensus.cc:482] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:28.520118 29885 raft_consensus.cc:3037] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:28.521193 29885 raft_consensus.cc:504] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.521533 29885 leader_election.cc:288] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [CANDIDATE]: Term 1 election: Requested vote from peers c12a5e0e0764494aaa0e08a6285deeb8 (127.28.21.2:42931), f21df0b6b1104f3fac61b045290acd20 (127.28.21.3:45133)
I20260806 02:07:28.521862 29669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d6be7271b44db6870bafb6eed32dd3" candidate_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12a5e0e0764494aaa0e08a6285deeb8"
I20260806 02:07:28.521898 29810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d6be7271b44db6870bafb6eed32dd3" candidate_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21df0b6b1104f3fac61b045290acd20"
I20260806 02:07:28.521996 29669 raft_consensus.cc:3037] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:28.522047 29810 raft_consensus.cc:3037] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:28.523128 29669 raft_consensus.cc:2454] T f4d6be7271b44db6870bafb6eed32dd3 P c12a5e0e0764494aaa0e08a6285deeb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b7cfbe81cd4615adbaf61d7b046b88 in term 1.
I20260806 02:07:28.523175 29810 raft_consensus.cc:2454] T f4d6be7271b44db6870bafb6eed32dd3 P f21df0b6b1104f3fac61b045290acd20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b7cfbe81cd4615adbaf61d7b046b88 in term 1.
I20260806 02:07:28.523466 29453 leader_election.cc:302] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b7cfbe81cd4615adbaf61d7b046b88, c12a5e0e0764494aaa0e08a6285deeb8; no voters:
I20260806 02:07:28.523701 29885 raft_consensus.cc:2781] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:28.523885 29885 raft_consensus.cc:686] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [term 1 LEADER]: Becoming Leader. State: Replica: 56b7cfbe81cd4615adbaf61d7b046b88, State: Running, Role: LEADER
I20260806 02:07:28.524422 29885 consensus_queue.cc:237] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } }
I20260806 02:07:28.528028 29368 catalog_manager.cc:5495] T f4d6be7271b44db6870bafb6eed32dd3 P 56b7cfbe81cd4615adbaf61d7b046b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56b7cfbe81cd4615adbaf61d7b046b88 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12a5e0e0764494aaa0e08a6285deeb8" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21df0b6b1104f3fac61b045290acd20" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 45133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56b7cfbe81cd4615adbaf61d7b046b88" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42837 } health_report { overall_health: HEALTHY } } }
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:28.555588 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.546515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.28.21.1:42837)
0806 02:07:28.546697 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.546702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.546923 (+ 221us) server_negotiation.cc:407] Connection header received
0806 02:07:28.547096 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.547099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.547148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.547240 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.548060 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.548574 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.549417 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.549654 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.549763 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.552265 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.552278 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.552298 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.552326 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.553826 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.554253 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.554259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.554261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.554345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.554642 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.554646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.554647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.554893 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.555119 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.555279 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.555400 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:28.582767 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.574691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.28.21.1:42837)
0806 02:07:28.574824 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.574828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.574979 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:07:28.575097 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.575100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.575140 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.575213 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.575872 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.576365 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.577161 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.577377 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.577449 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.579802 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.579805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.579807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.579830 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.581261 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.581678 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.581681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.581682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.581755 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.581988 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.581991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.581992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.582146 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.582261 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.582550 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.582645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:28.609301 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:28.601090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.28.21.1:42837)
0806 02:07:28.601255 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:28.601258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:28.601307 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:07:28.601429 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:28.601432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:28.601494 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:28.601566 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:28.602223 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.602792 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.603622 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:28.603841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:28.603964 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:28.606669 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:28.606672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:28.606674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:28.606700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:28.607888 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.608263 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.608266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.608267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.608341 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:28.608588 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:28.608590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:28.608591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:28.608750 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:28.608868 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:28.609057 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 02:07:28.609164 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:07:28.611032 29485 tablet_service.cc:2908] Rejecting scan request for tablet f4d6be7271b44db6870bafb6eed32dd3: Uninitialized: clean time has not yet been initialized
I20260806 02:07:28.613373 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29435
I20260806 02:07:28.620918 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29580
I20260806 02:07:28.627102 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29721
I20260806 02:07:28.633412 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29332
2026-08-06T02:07:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2379 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29904](info): set up 2 sockets
Aug 06 02:07:28 dist-test-slave-gp6n krb5kdc[29904](info): commencing operation
krb5kdc: starting...
W20260806 02:07:29.761323 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:29 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:29.887552 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41325
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:29.999995 29918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:30.000255 29918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:30.000295 29918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:30.004284 29918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:30.004338 29918 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:07:30.004355 29918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:30.004410 29918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:30.004448 29918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:30.004469 29918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:30.009876 29918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:30.010985 29918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:30.012030 29918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:30.020418 29927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.020418 29925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.020573 29918 server_base.cc:1034] running on GCE node
W20260806 02:07:30.020707 29924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.021231 29918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:30.022302 29918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:30.023483 29918 hybrid_clock.cc:648] HybridClock initialized: now 1785982050023472 us; error 24 us; skew 500 ppm
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:30.026216 29918 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:30.027179 29918 webserver.cc:460] Webserver started at http://127.28.21.62:36573/ using document root <none> and password file <none>
I20260806 02:07:30.027729 29918 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:30.027815 29918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:30.028039 29918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:30.030099 29918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "7070b20bfb6541b296ea89a687073b02"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "04ceb9cf653a43694f587a38165ccaac"
server_key_iv: "1879c8b4634f4071dd3098bf11114d4a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.030699 29918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "7070b20bfb6541b296ea89a687073b02"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "04ceb9cf653a43694f587a38165ccaac"
server_key_iv: "1879c8b4634f4071dd3098bf11114d4a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.034283 29918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:07:30.036618 29934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.037854 29918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:30.038012 29918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "7070b20bfb6541b296ea89a687073b02"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "04ceb9cf653a43694f587a38165ccaac"
server_key_iv: "1879c8b4634f4071dd3098bf11114d4a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.038125 29918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:30.055562 29918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:30.058825 29918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:30.059043 29918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:30.067731 29918 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:41325
I20260806 02:07:30.067761 29999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:41325 every 8 connection(s)
I20260806 02:07:30.068694 29918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:30.071589 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:30.074213 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 29918
I20260806 02:07:30.074349 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:30.074666 28756 external_mini_cluster.cc:1428] Setting key 2ee493e54f106943657250123c76e086
I20260806 02:07:30.077950 30000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Bootstrap starting.
I20260806 02:07:30.080269 30000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:30.080996 30000 log.cc:824] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:30.083021 30000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: No bootstrap required, opened a new log
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:30.086686 30000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } }
I20260806 02:07:30.086962 30000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:30.087052 30000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7070b20bfb6541b296ea89a687073b02, State: Initialized, Role: FOLLOWER
I20260806 02:07:30.087545 30000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } }
I20260806 02:07:30.087699 30000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:30.087774 30000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:30.087900 30000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:30.088922 30000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } }
I20260806 02:07:30.089180 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.076156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.28.21.62:41325)
0806 02:07:30.076640 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.076655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.076691 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:07:30.077312 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.077329 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.077668 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.078008 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.078865 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.079874 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.080797 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.081336 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.081519 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.084344 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.084349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.084362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.084389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.086152 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.086685 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.086690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.086695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.086811 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.087126 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.087128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.087129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.087561 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.087789 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.088078 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.088358 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":144,"threads_started":1}
I20260806 02:07:30.089339 30000 leader_election.cc:302] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7070b20bfb6541b296ea89a687073b02; no voters:
I20260806 02:07:30.089700 30000 leader_election.cc:288] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:30.089807 30005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:30.090016 30005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [term 1 LEADER]: Becoming Leader. State: Replica: 7070b20bfb6541b296ea89a687073b02, State: Running, Role: LEADER
I20260806 02:07:30.090322 30005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } }
I20260806 02:07:30.090960 30000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:30.091560 30005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7070b20bfb6541b296ea89a687073b02. Latest consensus state: current_term: 1 leader_uuid: "7070b20bfb6541b296ea89a687073b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } } }
I20260806 02:07:30.091670 30005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:30.092274 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7070b20bfb6541b296ea89a687073b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7070b20bfb6541b296ea89a687073b02" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41325 } } }
I20260806 02:07:30.092374 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:30.092651 30010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:30.095486 30010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:30.101725 30010 catalog_manager.cc:1347] Generated new cluster ID: b3fbcd9e0e724982b4aee8b4037281dd
I20260806 02:07:30.101794 30010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:30.116931 30010 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:30.117868 30010 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:30.130107 30010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Generated new TSK 0
I20260806 02:07:30.130986 30010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:30.167166 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41325
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:30.285534 30027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:30.285794 30027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:30.285883 30027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:30.289929 30027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:30.289986 30027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:30.290063 30027 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:30.290145 30027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:30.295663 30027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:41325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:30.296713 30027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:30.297768 30027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:30.307821 30032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.307812 30033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.307794 30035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.308193 30027 server_base.cc:1034] running on GCE node
I20260806 02:07:30.308651 30027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:30.309245 30027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:30.310483 30027 hybrid_clock.cc:648] HybridClock initialized: now 1785982050310453 us; error 32 us; skew 500 ppm
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:30.317911 30027 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:30.318938 30027 webserver.cc:460] Webserver started at http://127.28.21.1:46089/ using document root <none> and password file <none>
I20260806 02:07:30.319548 30027 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:30.319644 30027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:30.319883 30027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:30.321933 30027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "0391128c25894e06bc0604df4da36b59"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "9a753fcb89d4cfc5197c11dc4ad812a6"
server_key_iv: "01ecf5540efad5e5fcb46cb1e8e3552a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.322583 30027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "0391128c25894e06bc0604df4da36b59"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "9a753fcb89d4cfc5197c11dc4ad812a6"
server_key_iv: "01ecf5540efad5e5fcb46cb1e8e3552a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.326325 30027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:30.328841 30042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.330081 30027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:07:30.330220 30027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "0391128c25894e06bc0604df4da36b59"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "9a753fcb89d4cfc5197c11dc4ad812a6"
server_key_iv: "01ecf5540efad5e5fcb46cb1e8e3552a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.330336 30027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:30.356026 30027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:30.359436 30027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:30.359655 30027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:30.361107 30027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:30.361181 30027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.361254 30027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:30.361299 30027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:30.372581 30027 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:36345
I20260806 02:07:30.372643 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:36345 every 8 connection(s)
I20260806 02:07:30.373747 30027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:30.375635 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30027
I20260806 02:07:30.375782 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:30.376109 28756 external_mini_cluster.cc:1428] Setting key b05f15e1a3fee5ef33563bf660f2388c
I20260806 02:07:30.376137 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.362203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:56261 (local address 127.28.21.62:41325)
0806 02:07:30.362373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.362377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.363486 (+ 1109us) server_negotiation.cc:407] Connection header received
0806 02:07:30.364725 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.364729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.364790 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.364906 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.366481 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.367406 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.368800 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.369233 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.369377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.372110 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.372114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.372117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.372143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.373924 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.374597 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.374601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.374603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.374704 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.375045 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.375048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.375049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.375229 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.375308 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.375812 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.375952 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:07:30.377002 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.362767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.1:56261)
0806 02:07:30.363287 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.363370 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.364494 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.365097 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.365106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.365531 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.366242 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.366261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.367573 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.367577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.368685 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.368695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.369418 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.369423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.369576 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.370207 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.370231 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.371929 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.374045 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.374050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.374061 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.374464 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.374808 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.374811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.374812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.374942 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.375317 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.375321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.375545 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.375689 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.376019 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":155,"threads_started":1}
I20260806 02:07:30.378274 30163 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41325
I20260806 02:07:30.378443 30163 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:30.378922 30163 heartbeater.cc:511] Master 127.28.21.62:41325 requested a full tablet report, sending...
I20260806 02:07:30.380468 29951 ts_manager.cc:194] Registered new tserver with Master: 0391128c25894e06bc0604df4da36b59 (127.28.21.1:36345)
I20260806 02:07:30.381786 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:56261
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:30.389839 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.380834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.1:58103)
0806 02:07:30.381100 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.381114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.381201 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.381444 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.381447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.381672 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.381818 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.381828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.383007 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.383013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.384048 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.384061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.384556 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.384564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.384773 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.385485 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.385521 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.386246 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.388066 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.388069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.388076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.388521 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.388826 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.388828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.388829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.388941 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.389319 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.389321 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.389491 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.389587 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.389695 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
I20260806 02:07:30.389851 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.381119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:58103 (local address 127.28.21.62:41325)
0806 02:07:30.381232 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.381238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.381264 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:07:30.381299 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.381302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.381345 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.381426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.381960 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.382823 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.384183 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.384389 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.384502 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.386511 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.386514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.386515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.386536 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.387913 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.388645 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.388648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.388649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.388721 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.389034 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.389036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.389037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.389193 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.389270 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.389654 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.389747 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:30.409789 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41325
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:30.521946 30172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:30.522238 30172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:30.522282 30172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:30.526252 30172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:30.526314 30172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:30.526352 30172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:30.526384 30172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:30.531554 30172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:41325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:30.532524 30172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:30.533533 30172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:30.543195 30180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.543207 30182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.543216 30179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.543437 30172 server_base.cc:1034] running on GCE node
I20260806 02:07:30.543898 30172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:30.544597 30172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:30.545778 30172 hybrid_clock.cc:648] HybridClock initialized: now 1785982050545772 us; error 39 us; skew 500 ppm
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:30.548473 30172 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:30.549311 30172 webserver.cc:460] Webserver started at http://127.28.21.2:36263/ using document root <none> and password file <none>
I20260806 02:07:30.549880 30172 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:30.549971 30172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:30.550200 30172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:30.552281 30172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "ded7300cb8ba456081626002bcbb5ba3"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "2ea780238386438613bcd6fcbc619bb4"
server_key_iv: "670a284c08f7228fcbe0e385e79bd3a7"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.552846 30172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "ded7300cb8ba456081626002bcbb5ba3"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "2ea780238386438613bcd6fcbc619bb4"
server_key_iv: "670a284c08f7228fcbe0e385e79bd3a7"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.556907 30172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:30.559228 30190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.560338 30172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:30.560472 30172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "ded7300cb8ba456081626002bcbb5ba3"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "2ea780238386438613bcd6fcbc619bb4"
server_key_iv: "670a284c08f7228fcbe0e385e79bd3a7"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.560578 30172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:30.593832 30172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:30.596896 30172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:30.597103 30172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:30.598611 30172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:30.598681 30172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.598750 30172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:30.598780 30172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:30.609722 30172 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:41891
I20260806 02:07:30.609841 30311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:41891 every 8 connection(s)
I20260806 02:07:30.610870 30172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:30.613624 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.599424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:35683 (local address 127.28.21.62:41325)
0806 02:07:30.599551 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.599554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.600608 (+ 1054us) server_negotiation.cc:407] Connection header received
0806 02:07:30.601654 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.601659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.601716 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.601781 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.603547 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.604457 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.606258 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.606676 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.606796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.609721 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.609725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.609728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.609763 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.611428 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.612080 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.612085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.612087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.612186 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.612540 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.612544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.612545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.612734 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.612847 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.613327 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.613450 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:07:30.614337 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.599983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.2:35683)
0806 02:07:30.600461 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.600502 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.601393 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.602259 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.602271 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.602760 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.603363 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.603384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.604576 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.604579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.606137 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.606150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.606821 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.606836 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.606993 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.607585 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.607611 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.609537 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.611592 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.611597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.611611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.611965 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.612284 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.612291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.612294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.612438 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.612849 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.612855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.613089 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.613273 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.613574 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260806 02:07:30.615559 30312 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41325
I20260806 02:07:30.615698 30312 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:30.616227 30312 heartbeater.cc:511] Master 127.28.21.62:41325 requested a full tablet report, sending...
I20260806 02:07:30.616531 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30172
I20260806 02:07:30.616659 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:30.616940 28756 external_mini_cluster.cc:1428] Setting key 048daa09a9ac69ac3996fcd6964bb19e
I20260806 02:07:30.617853 29953 ts_manager.cc:194] Registered new tserver with Master: ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891)
I20260806 02:07:30.618672 29953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:35683
I20260806 02:07:30.625663 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.617868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:60251 (local address 127.28.21.62:41325)
0806 02:07:30.617965 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.617970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.618153 (+ 183us) server_negotiation.cc:407] Connection header received
0806 02:07:30.618298 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.618301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.618343 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.618389 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.618985 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.619771 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.620779 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.621158 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.621279 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.622496 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.622500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.622502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.622532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.623946 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.624456 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.624458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.624459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.624529 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.624854 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.624857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.624858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.625053 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.625132 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.625468 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.625558 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:07:30.625808 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.617696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.2:60251)
0806 02:07:30.618065 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.618080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.618185 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.618540 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.618543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.618745 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.618857 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.618864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.619915 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.619918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.620660 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.620669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.621269 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.621272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.621335 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.621845 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.621864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.622321 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.624059 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.624066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.624071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.624354 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.624624 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.624630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.624634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.624742 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.625153 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.625159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.625279 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.625404 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.625571 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:30.645699 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41325
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:30.757978 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:30.758296 30321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:30.758394 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:30.762645 30321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:30.762733 30321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:30.762801 30321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:30.762887 30321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:30.768435 30321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:41325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:30.769593 30321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:30.770660 30321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:30.780431 30327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:30.780500 30329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.780615 30321 server_base.cc:1034] running on GCE node
W20260806 02:07:30.780458 30326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:30.781221 30321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:30.781864 30321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:30.783034 30321 hybrid_clock.cc:648] HybridClock initialized: now 1785982050783010 us; error 27 us; skew 500 ppm
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:30.785827 30321 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:30.786816 30321 webserver.cc:460] Webserver started at http://127.28.21.3:33177/ using document root <none> and password file <none>
I20260806 02:07:30.787375 30321 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:30.787467 30321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:30.787703 30321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:30.789727 30321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "53466dff056243aaaf5b9aad8c4f90af"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "497fd8cd5a70b81ad0316681d1a15775"
server_key_iv: "2e5facd56d97c6699bd2492cd9dd7d1a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.790297 30321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "53466dff056243aaaf5b9aad8c4f90af"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "497fd8cd5a70b81ad0316681d1a15775"
server_key_iv: "2e5facd56d97c6699bd2492cd9dd7d1a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.794530 30321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:07:30.797009 30337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.798218 30321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:30.798378 30321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "53466dff056243aaaf5b9aad8c4f90af"
format_stamp: "Formatted at 2026-08-06 02:07:30 on dist-test-slave-gp6n"
server_key: "497fd8cd5a70b81ad0316681d1a15775"
server_key_iv: "2e5facd56d97c6699bd2492cd9dd7d1a"
server_key_version: "encryptionkey@0"
I20260806 02:07:30.798557 30321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:30.812425 30321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:30.815212 30321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:30.815404 30321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:30.817039 30321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:30.817099 30321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:30.817150 30321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:30.817178 30321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:30.828305 30321 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:34689
I20260806 02:07:30.828405 30459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:34689 every 8 connection(s)
I20260806 02:07:30.829250 30321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:30.831053 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.817809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:37993 (local address 127.28.21.62:41325)
0806 02:07:30.817962 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.817967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.819539 (+ 1572us) server_negotiation.cc:407] Connection header received
0806 02:07:30.820831 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.820835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.820895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.820989 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.822296 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.822901 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.824110 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.824349 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.824470 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.827351 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.827354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.827356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.827380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.828937 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.829516 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.829519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.829521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.829599 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.830017 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.830020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.830022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.830231 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.830312 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.830820 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.830936 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:07:30.831773 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.818826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.3:37993)
0806 02:07:30.819348 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.819412 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.820575 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.821155 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.821162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.821640 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.822141 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.822154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.823032 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.823040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.823975 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.823992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.824518 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.824522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.824681 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.825256 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.825272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.827180 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.829048 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.829053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.829066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.829417 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.829726 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.829731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.829733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.829903 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.830342 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.830347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.830613 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.830768 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.831002 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260806 02:07:30.832177 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30321
I20260806 02:07:30.832311 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:30.832641 28756 external_mini_cluster.cc:1428] Setting key 6355f2e7705a9230fa1b4cabfb8b7d5f
I20260806 02:07:30.833107 30460 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41325
I20260806 02:07:30.833276 30460 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:30.833896 30460 heartbeater.cc:511] Master 127.28.21.62:41325 requested a full tablet report, sending...
I20260806 02:07:30.834970 29951 ts_manager.cc:194] Registered new tserver with Master: 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:30.835693 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:37993
I20260806 02:07:30.842003 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.835255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:56385 (local address 127.28.21.62:41325)
0806 02:07:30.835387 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.835391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.835564 (+ 173us) server_negotiation.cc:407] Connection header received
0806 02:07:30.835656 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.835658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.835692 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.835754 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.836220 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.836766 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.837574 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.837780 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.837888 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.838944 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.838948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.838950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.838982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.840379 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.840825 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.840828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.840831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.840909 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.841183 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.841187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.841188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.841353 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.841455 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.841742 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.841844 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:07:30.842034 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.835205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.28.21.3:56385)
0806 02:07:30.835482 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.835495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.835582 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.835795 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.835798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.836003 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:30.836113 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.836119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.836898 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.836901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.837457 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.837464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.837891 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.837895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.837983 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.838477 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:30.838497 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:30.838802 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:30.840527 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.840530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.840532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.840724 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.841000 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:30.841004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:30.841006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:30.841092 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:30.841459 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:30.841463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:30.841555 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.841677 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.841810 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
I20260806 02:07:30.846488 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:07:30 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:30.872239 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.863914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.28.21.62:41325)
0806 02:07:30.864060 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.864063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.864076 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:30.864112 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.864114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.864150 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.864220 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:30.864984 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.865501 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.866290 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.866585 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.866676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.869008 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:30.869011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:30.869013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:30.869036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:30.870548 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.870981 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.870985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.870987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.871071 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:30.871362 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:30.871366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:30.871367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:30.871544 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:30.871680 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.871950 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.872063 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:07:30.875272 29951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:30.878317 29951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:30.892248 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.886348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:41891 (local address 127.0.0.1:33446)
0806 02:07:30.886755 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.886810 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.886950 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.887894 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.887900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.887920 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:30.888062 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.888069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.889372 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.889379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.890536 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.890547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.891676 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.891681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.891797 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.891837 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.891952 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.892089 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1,"thread_start_us":149,"threads_started":1}
I20260806 02:07:30.892256 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.886566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.28.21.2:41891)
0806 02:07:30.886980 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.886992 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.887020 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:07:30.887101 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.887107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.887502 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.888210 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:30.888357 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.889249 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.890669 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.891549 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.891654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.891835 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.892002 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.892101 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":104,"threads_started":1}
I20260806 02:07:30.893158 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.887148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36345 (local address 127.0.0.1:51458)
0806 02:07:30.887663 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.887677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.887758 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.888379 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.888384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.888406 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:30.888585 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.888592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.890103 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.890106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.891298 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.891308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.892693 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.892696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.892829 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.892851 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.892927 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.893002 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":91,"thread_start_us":93,"threads_started":1}
I20260806 02:07:30.893296 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.886805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:34689 (local address 127.0.0.1:45916)
0806 02:07:30.887315 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:30.887331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:30.887418 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:30.888432 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:30.888435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:30.888447 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:30.888654 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.888662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.890038 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.890042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.891281 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:30.891291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.892603 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.892607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:30.892995 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.893016 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:07:30.893080 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:07:30.893136 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":389,"thread_start_us":96,"threads_started":1}
I20260806 02:07:30.893759 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.887223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.28.21.3:34689)
0806 02:07:30.888014 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.888029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.888052 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:07:30.888126 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.888130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.888269 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.888552 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:30.888775 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.889915 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.891437 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.892372 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.892575 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.892733 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.893471 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.893575 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":138,"threads_started":1}
I20260806 02:07:30.894172 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:30.887565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.28.21.1:36345)
0806 02:07:30.887944 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:30.887964 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:30.887987 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:07:30.888060 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:30.888064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:30.888246 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:30.888380 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:30.888738 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.889915 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.891525 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:30.892350 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:30.892470 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:30.892636 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:30.893951 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0806 02:07:30.894052 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":92,"threads_started":1}
I20260806 02:07:30.896042 30389 tablet_service.cc:1467] Processing CreateTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 (DEFAULT_TABLE table=test-table [id=a95e7b154f5447d5a80c5e58a04f0297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:30.896052 30243 tablet_service.cc:1467] Processing CreateTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 (DEFAULT_TABLE table=test-table [id=a95e7b154f5447d5a80c5e58a04f0297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:30.897009 30389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d72cec16be4b869fccb1025f8ccbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:30.897068 30243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d72cec16be4b869fccb1025f8ccbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:30.896793 30095 tablet_service.cc:1467] Processing CreateTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 (DEFAULT_TABLE table=test-table [id=a95e7b154f5447d5a80c5e58a04f0297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:30.897673 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d72cec16be4b869fccb1025f8ccbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:30.903843 30483 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Bootstrap starting.
I20260806 02:07:30.905155 30484 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Bootstrap starting.
I20260806 02:07:30.906392 30483 tablet_bootstrap.cc:654] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:30.906739 30485 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Bootstrap starting.
I20260806 02:07:30.907186 30484 tablet_bootstrap.cc:654] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:30.907305 30483 log.cc:824] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:30.907970 30484 log.cc:824] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:30.908715 30485 tablet_bootstrap.cc:654] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:30.909520 30485 log.cc:824] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:30.909582 30483 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: No bootstrap required, opened a new log
I20260806 02:07:30.909749 30483 ts_tablet_manager.cc:1397] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:07:30.910346 30484 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: No bootstrap required, opened a new log
I20260806 02:07:30.910569 30484 ts_tablet_manager.cc:1397] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:07:30.911414 30485 tablet_bootstrap.cc:492] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: No bootstrap required, opened a new log
I20260806 02:07:30.911590 30485 ts_tablet_manager.cc:1397] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:30.913646 30483 raft_consensus.cc:348] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.913949 30483 raft_consensus.cc:374] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:30.914018 30483 raft_consensus.cc:729] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53466dff056243aaaf5b9aad8c4f90af, State: Initialized, Role: FOLLOWER
I20260806 02:07:30.913838 30484 raft_consensus.cc:348] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.914086 30484 raft_consensus.cc:374] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:30.914141 30484 raft_consensus.cc:729] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0391128c25894e06bc0604df4da36b59, State: Initialized, Role: FOLLOWER
I20260806 02:07:30.914549 30484 consensus_queue.cc:260] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.914520 30483 consensus_queue.cc:260] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.915357 30485 raft_consensus.cc:348] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.915560 30483 ts_tablet_manager.cc:1428] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:07:30.915568 30484 ts_tablet_manager.cc:1428] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:07:30.915593 30485 raft_consensus.cc:374] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:30.915645 30485 raft_consensus.cc:729] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded7300cb8ba456081626002bcbb5ba3, State: Initialized, Role: FOLLOWER
I20260806 02:07:30.915934 30460 heartbeater.cc:503] Master 127.28.21.62:41325 was elected leader, sending a full tablet report...
I20260806 02:07:30.916056 30163 heartbeater.cc:503] Master 127.28.21.62:41325 was elected leader, sending a full tablet report...
I20260806 02:07:30.915995 30485 consensus_queue.cc:260] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:30.916604 30312 heartbeater.cc:503] Master 127.28.21.62:41325 was elected leader, sending a full tablet report...
I20260806 02:07:30.916919 30485 ts_tablet_manager.cc:1428] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:07:31.030172 30490 raft_consensus.cc:482] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:31.030620 30490 raft_consensus.cc:504] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.031961 30490 leader_election.cc:288] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.037657 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.032603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:47149 (local address 127.28.21.3:34689)
0806 02:07:31.032732 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.032737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.032966 (+ 229us) server_negotiation.cc:407] Connection header received
0806 02:07:31.033123 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.033126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.033181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.033277 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:31.033657 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.034637 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.036370 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.036970 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.037082 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.037144 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.037455 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.037526 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:31.037657 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.032603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:51669 (local address 127.28.21.2:41891)
0806 02:07:31.032732 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.032737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.033030 (+ 293us) server_negotiation.cc:407] Connection header received
0806 02:07:31.033122 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.033126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.033181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.033277 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:31.033657 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.034510 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.036355 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.036970 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.037082 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.037144 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.037455 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.037526 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:07:31.037744 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.032424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:41891 (local address 127.28.21.1:51669)
0806 02:07:31.032939 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.032950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.033027 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.033343 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.033347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.033393 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:31.033540 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.033547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.034654 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.034657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.036213 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.036225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.037100 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.037104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.037291 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.037345 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.037422 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.037503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":198,"spinlock_wait_cycles":27264,"thread_start_us":124,"threads_started":1}
I20260806 02:07:31.037744 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.032404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:34689 (local address 127.28.21.1:47149)
0806 02:07:31.032831 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.032849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.033002 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.033343 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.033347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.033394 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:31.033540 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.033547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.034785 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.034788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.036216 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.036225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.037100 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.037104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.037266 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.037338 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.037422 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.037503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":136,"spinlock_wait_cycles":10368,"thread_start_us":139,"threads_started":1}
I20260806 02:07:31.038487 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d72cec16be4b869fccb1025f8ccbd3" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7300cb8ba456081626002bcbb5ba3" is_pre_election: true
I20260806 02:07:31.038848 30263 raft_consensus.cc:2454] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 0.
I20260806 02:07:31.038810 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d72cec16be4b869fccb1025f8ccbd3" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af" is_pre_election: true
I20260806 02:07:31.039152 30410 raft_consensus.cc:2454] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 0.
I20260806 02:07:31.039369 30043 leader_election.cc:302] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, ded7300cb8ba456081626002bcbb5ba3; no voters:
I20260806 02:07:31.039644 30490 raft_consensus.cc:2781] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:31.039731 30490 raft_consensus.cc:482] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:31.039778 30490 raft_consensus.cc:3037] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.040946 30490 raft_consensus.cc:504] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.041365 30490 leader_election.cc:288] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 election: Requested vote from peers ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.041718 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d72cec16be4b869fccb1025f8ccbd3" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af"
I20260806 02:07:31.041728 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d72cec16be4b869fccb1025f8ccbd3" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7300cb8ba456081626002bcbb5ba3"
I20260806 02:07:31.041873 30410 raft_consensus.cc:3037] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.041901 30263 raft_consensus.cc:3037] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.043814 30410 raft_consensus.cc:2454] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 1.
I20260806 02:07:31.044114 30046 leader_election.cc:302] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, 53466dff056243aaaf5b9aad8c4f90af; no voters:
I20260806 02:07:31.044272 30490 raft_consensus.cc:2781] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:31.044483 30490 raft_consensus.cc:686] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 1 LEADER]: Becoming Leader. State: Replica: 0391128c25894e06bc0604df4da36b59, State: Running, Role: LEADER
I20260806 02:07:31.044783 30490 consensus_queue.cc:237] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.045476 30263 raft_consensus.cc:2454] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 1.
I20260806 02:07:31.048486 29950 catalog_manager.cc:5495] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0391128c25894e06bc0604df4da36b59 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } health_report { overall_health: UNKNOWN } } }
W20260806 02:07:31.080394 30461 tablet.cc:2369] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:31.089570 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.085641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.28.21.1:36345)
0806 02:07:31.085792 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.085797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.085813 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:07:31.086037 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.086039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.086102 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.086205 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:31.086535 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.087059 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.087987 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.088208 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.088314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.088676 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.089115 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.089247 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.089352 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.089421 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:07:31.096266 30410 raft_consensus.cc:1261] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Refusing update from remote peer 0391128c25894e06bc0604df4da36b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.097015 30490 consensus_queue.cc:1035] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.098026 30263 raft_consensus.cc:1261] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Refusing update from remote peer 0391128c25894e06bc0604df4da36b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.098642 30490 consensus_queue.cc:1035] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.104331 30498 mvcc.cc:205] Tried to move back new op lower bound from 7315382481281556480 to 7315382481083342848. Current Snapshot: MvccSnapshot[applied={T|T < 7315382481281556480}]
I20260806 02:07:31.119699 29950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:31.119997 29950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:31.122706 29950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Sending DeleteTablet for 3 replicas of tablet 30d72cec16be4b869fccb1025f8ccbd3
I20260806 02:07:31.123649 30389 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:45916
I20260806 02:07:31.123646 30095 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51458
I20260806 02:07:31.124616 29950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:31.124969 29950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:31.126348 30243 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d72cec16be4b869fccb1025f8ccbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33446
I20260806 02:07:31.126596 30507 tablet_replica.cc:330] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: stopping tablet replica
I20260806 02:07:31.126823 30509 tablet_replica.cc:330] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: stopping tablet replica
I20260806 02:07:31.126925 30507 raft_consensus.cc:2229] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:31.127077 30509 raft_consensus.cc:2229] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:31.127339 30507 raft_consensus.cc:2258] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.127377 30509 raft_consensus.cc:2258] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.128986 30507 ts_tablet_manager.cc:1905] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.130141 30095 tablet_service.cc:1467] Processing CreateTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 (DEFAULT_TABLE table=test-table [id=3485c972d17c438db50a1611e2a82f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:31.130453 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb579aa7bf8e40d78433f0c0ca7829a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.130578 30389 tablet_service.cc:1467] Processing CreateTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 (DEFAULT_TABLE table=test-table [id=3485c972d17c438db50a1611e2a82f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:31.130875 30389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb579aa7bf8e40d78433f0c0ca7829a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.131208 30509 ts_tablet_manager.cc:1905] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.133183 30484 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Bootstrap starting.
I20260806 02:07:31.133189 30483 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Bootstrap starting.
I20260806 02:07:31.134097 30509 ts_tablet_manager.cc:1918] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:31.134217 30484 tablet_bootstrap.cc:654] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.134214 30509 log.cc:1196] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/30d72cec16be4b869fccb1025f8ccbd3
I20260806 02:07:31.134248 30483 tablet_bootstrap.cc:654] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.134603 30509 ts_tablet_manager.cc:1939] T 30d72cec16be4b869fccb1025f8ccbd3 P ded7300cb8ba456081626002bcbb5ba3: Deleting consensus metadata
I20260806 02:07:31.135070 30243 tablet_service.cc:1467] Processing CreateTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 (DEFAULT_TABLE table=test-table [id=3485c972d17c438db50a1611e2a82f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:31.135305 30243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb579aa7bf8e40d78433f0c0ca7829a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.135617 30484 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: No bootstrap required, opened a new log
I20260806 02:07:31.135694 30484 ts_tablet_manager.cc:1397] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:31.135912 29936 catalog_manager.cc:4841] TS ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891): tablet 30d72cec16be4b869fccb1025f8ccbd3 (table test-table [id=a95e7b154f5447d5a80c5e58a04f0297]) successfully deleted
I20260806 02:07:31.136117 30484 raft_consensus.cc:348] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.136220 30484 raft_consensus.cc:374] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.136250 30484 raft_consensus.cc:729] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0391128c25894e06bc0604df4da36b59, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.136345 30484 consensus_queue.cc:260] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.136613 30484 ts_tablet_manager.cc:1428] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:31.137569 30485 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Bootstrap starting.
I20260806 02:07:31.138497 30485 tablet_bootstrap.cc:654] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.140025 30507 ts_tablet_manager.cc:1918] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:31.140105 30507 log.cc:1196] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/30d72cec16be4b869fccb1025f8ccbd3
I20260806 02:07:31.140403 30507 ts_tablet_manager.cc:1939] T 30d72cec16be4b869fccb1025f8ccbd3 P 0391128c25894e06bc0604df4da36b59: Deleting consensus metadata
I20260806 02:07:31.141546 29936 catalog_manager.cc:4841] TS 0391128c25894e06bc0604df4da36b59 (127.28.21.1:36345): tablet 30d72cec16be4b869fccb1025f8ccbd3 (table test-table [id=a95e7b154f5447d5a80c5e58a04f0297]) successfully deleted
I20260806 02:07:31.141609 30508 tablet_replica.cc:330] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: stopping tablet replica
I20260806 02:07:31.141979 30508 raft_consensus.cc:2229] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:31.142328 30508 raft_consensus.cc:2258] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.142625 30483 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: No bootstrap required, opened a new log
I20260806 02:07:31.142719 30483 ts_tablet_manager.cc:1397] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260806 02:07:31.143313 30485 tablet_bootstrap.cc:492] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: No bootstrap required, opened a new log
I20260806 02:07:31.143265 30483 raft_consensus.cc:348] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.143385 30483 raft_consensus.cc:374] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.143406 30485 ts_tablet_manager.cc:1397] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260806 02:07:31.143445 30483 raft_consensus.cc:729] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53466dff056243aaaf5b9aad8c4f90af, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.143584 30483 consensus_queue.cc:260] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.143826 30483 ts_tablet_manager.cc:1428] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:31.143886 30485 raft_consensus.cc:348] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.144016 30485 raft_consensus.cc:374] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.144066 30485 raft_consensus.cc:729] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded7300cb8ba456081626002bcbb5ba3, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.144208 30485 consensus_queue.cc:260] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.144517 30485 ts_tablet_manager.cc:1428] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:07:31.144709 30508 ts_tablet_manager.cc:1905] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.148181 30508 ts_tablet_manager.cc:1918] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:31.148281 30508 log.cc:1196] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/30d72cec16be4b869fccb1025f8ccbd3
I20260806 02:07:31.148612 30508 ts_tablet_manager.cc:1939] T 30d72cec16be4b869fccb1025f8ccbd3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting consensus metadata
I20260806 02:07:31.149797 29939 catalog_manager.cc:4841] TS 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689): tablet 30d72cec16be4b869fccb1025f8ccbd3 (table test-table [id=a95e7b154f5447d5a80c5e58a04f0297]) successfully deleted
I20260806 02:07:31.168781 30492 raft_consensus.cc:482] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:31.168922 30492 raft_consensus.cc:504] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.170075 30492 leader_election.cc:288] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0391128c25894e06bc0604df4da36b59 (127.28.21.1:36345), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.174523 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.170299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36345 (local address 127.28.21.2:40721)
0806 02:07:31.170701 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.170715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.170897 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.171213 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.171219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.171239 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:31.171377 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.171383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.172461 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.172464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.173284 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.173293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.174048 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.174053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.174166 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.174192 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.174271 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.174340 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":442,"threads_started":1}
I20260806 02:07:31.174561 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.170618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:40721 (local address 127.28.21.1:36345)
0806 02:07:31.170739 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.170743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.170800 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:07:31.171034 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.171039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.171094 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.171181 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:31.171506 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.172297 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.173418 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.173900 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.174008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.174075 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.174305 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.174378 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:07:31.175355 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb579aa7bf8e40d78433f0c0ca7829a3" candidate_uuid: "ded7300cb8ba456081626002bcbb5ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0391128c25894e06bc0604df4da36b59" is_pre_election: true
I20260806 02:07:31.175540 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.170499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:42325 (local address 127.28.21.3:34689)
0806 02:07:31.170633 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.170637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.170862 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:07:31.171093 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.171096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.171137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.171230 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:31.171567 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.172298 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.173869 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.174442 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.174539 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.174604 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.175341 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.175418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:07:31.175633 30116 raft_consensus.cc:2454] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded7300cb8ba456081626002bcbb5ba3 in term 0.
I20260806 02:07:31.175880 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.170325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:34689 (local address 127.28.21.2:42325)
0806 02:07:31.170787 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.170801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.170913 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.171314 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.171318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.171336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:31.171458 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.171465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.172422 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.172426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.173722 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.173734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.175088 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.175092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.175196 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.175219 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.175660 (+ 441us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.175738 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":36,"thread_start_us":130,"threads_started":1}
I20260806 02:07:31.176059 30191 leader_election.cc:302] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, ded7300cb8ba456081626002bcbb5ba3; no voters:
I20260806 02:07:31.176364 30492 raft_consensus.cc:2781] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:31.176347 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb579aa7bf8e40d78433f0c0ca7829a3" candidate_uuid: "ded7300cb8ba456081626002bcbb5ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af" is_pre_election: true
I20260806 02:07:31.176435 30492 raft_consensus.cc:482] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:31.176477 30492 raft_consensus.cc:3037] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.176505 30410 raft_consensus.cc:2454] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded7300cb8ba456081626002bcbb5ba3 in term 0.
I20260806 02:07:31.177767 30492 raft_consensus.cc:504] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.178146 30492 leader_election.cc:288] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [CANDIDATE]: Term 1 election: Requested vote from peers 0391128c25894e06bc0604df4da36b59 (127.28.21.1:36345), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.178519 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb579aa7bf8e40d78433f0c0ca7829a3" candidate_uuid: "ded7300cb8ba456081626002bcbb5ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af"
I20260806 02:07:31.178560 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb579aa7bf8e40d78433f0c0ca7829a3" candidate_uuid: "ded7300cb8ba456081626002bcbb5ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0391128c25894e06bc0604df4da36b59"
I20260806 02:07:31.178637 30410 raft_consensus.cc:3037] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.178689 30116 raft_consensus.cc:3037] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.179713 30116 raft_consensus.cc:2454] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded7300cb8ba456081626002bcbb5ba3 in term 1.
I20260806 02:07:31.179747 30410 raft_consensus.cc:2454] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded7300cb8ba456081626002bcbb5ba3 in term 1.
I20260806 02:07:31.180001 30191 leader_election.cc:302] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, ded7300cb8ba456081626002bcbb5ba3; no voters:
I20260806 02:07:31.180184 30492 raft_consensus.cc:2781] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:31.180390 30492 raft_consensus.cc:686] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 LEADER]: Becoming Leader. State: Replica: ded7300cb8ba456081626002bcbb5ba3, State: Running, Role: LEADER
I20260806 02:07:31.180636 30492 consensus_queue.cc:237] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.183028 29953 catalog_manager.cc:5495] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "ded7300cb8ba456081626002bcbb5ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:07:31 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:31.217412 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.208243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41325 (local address 127.0.0.1:43958)
0806 02:07:31.208573 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.208586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.208720 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.208987 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.208990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.209425 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:31.209521 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.209527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.210511 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.210517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.211084 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.211092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.211544 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.211548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.211631 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.212256 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:31.212268 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:31.213924 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:31.215790 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:31.215801 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:31.215808 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:31.216076 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:31.216407 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:31.216409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:31.216411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:31.216497 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:31.216963 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:31.216969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:31.217071 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.217161 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.217275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
I20260806 02:07:31.217705 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.208339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.28.21.62:41325)
0806 02:07:31.208503 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.208508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.208662 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:07:31.208836 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.208838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.208872 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.208955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:31.209638 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.210359 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.211203 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.211428 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.211542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.214070 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:31.214078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:31.214081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:31.214101 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:31.215687 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.216206 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.216209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.216211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.216311 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.216588 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.216591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.216593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.216839 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:31.216948 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.217250 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.217382 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:07:31.221727 29953 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:43958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:07:31.222615 29953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:31.227808 30095 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba5e8f04d424850b42d1c32dadb89e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3af06b07b12745c080e1ba43110bf928]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:31.228376 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba5e8f04d424850b42d1c32dadb89e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.230907 30243 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba5e8f04d424850b42d1c32dadb89e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3af06b07b12745c080e1ba43110bf928]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:31.231187 30484 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: Bootstrap starting.
I20260806 02:07:31.231309 30243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba5e8f04d424850b42d1c32dadb89e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.232431 30484 tablet_bootstrap.cc:654] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.233697 30484 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: No bootstrap required, opened a new log
I20260806 02:07:31.233791 30484 ts_tablet_manager.cc:1397] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:31.234265 30484 raft_consensus.cc:348] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.234472 30484 raft_consensus.cc:374] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.234513 30484 raft_consensus.cc:729] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0391128c25894e06bc0604df4da36b59, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.234673 30484 consensus_queue.cc:260] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.234987 30495 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.234930 30389 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba5e8f04d424850b42d1c32dadb89e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3af06b07b12745c080e1ba43110bf928]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:31.235126 30495 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.235318 30389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba5e8f04d424850b42d1c32dadb89e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:31.235772 30484 ts_tablet_manager.cc:1428] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:31.238361 30483 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: Bootstrap starting.
I20260806 02:07:31.238582 30485 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: Bootstrap starting.
I20260806 02:07:31.239233 30483 tablet_bootstrap.cc:654] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.239637 30485 tablet_bootstrap.cc:654] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:31.240558 30483 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: No bootstrap required, opened a new log
I20260806 02:07:31.240643 30483 ts_tablet_manager.cc:1397] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:31.240985 30485 tablet_bootstrap.cc:492] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: No bootstrap required, opened a new log
I20260806 02:07:31.240993 30483 raft_consensus.cc:348] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.241065 30485 ts_tablet_manager.cc:1397] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:07:31.241083 30483 raft_consensus.cc:374] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.241106 30483 raft_consensus.cc:729] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53466dff056243aaaf5b9aad8c4f90af, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.241269 30483 consensus_queue.cc:260] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.241566 30483 ts_tablet_manager.cc:1428] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:07:31.241498 30485 raft_consensus.cc:348] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.241524 30491 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.241608 30485 raft_consensus.cc:374] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:31.241668 30485 raft_consensus.cc:729] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded7300cb8ba456081626002bcbb5ba3, State: Initialized, Role: FOLLOWER
I20260806 02:07:31.241684 30491 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.241786 30485 consensus_queue.cc:260] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.242162 30492 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.242313 30492 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.242146 30485 ts_tablet_manager.cc:1428] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260806 02:07:31.362702 30313 tablet.cc:2369] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:07:31.375944 30164 tablet.cc:2369] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:31.430647 30495 raft_consensus.cc:482] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:31.430789 30495 raft_consensus.cc:504] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.431293 30495 leader_election.cc:288] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.431753 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba5e8f04d424850b42d1c32dadb89e5" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7300cb8ba456081626002bcbb5ba3" is_pre_election: true
I20260806 02:07:31.431835 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba5e8f04d424850b42d1c32dadb89e5" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af" is_pre_election: true
I20260806 02:07:31.431926 30263 raft_consensus.cc:2454] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 0.
I20260806 02:07:31.431993 30410 raft_consensus.cc:2454] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 0.
I20260806 02:07:31.432284 30043 leader_election.cc:302] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, 53466dff056243aaaf5b9aad8c4f90af, ded7300cb8ba456081626002bcbb5ba3; no voters:
I20260806 02:07:31.432475 30495 raft_consensus.cc:2781] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:31.432544 30495 raft_consensus.cc:482] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:31.432571 30495 raft_consensus.cc:3037] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.433799 30495 raft_consensus.cc:504] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.434175 30495 leader_election.cc:288] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 election: Requested vote from peers ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891), 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689)
I20260806 02:07:31.434556 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba5e8f04d424850b42d1c32dadb89e5" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7300cb8ba456081626002bcbb5ba3"
I20260806 02:07:31.434636 30410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba5e8f04d424850b42d1c32dadb89e5" candidate_uuid: "0391128c25894e06bc0604df4da36b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53466dff056243aaaf5b9aad8c4f90af"
I20260806 02:07:31.434711 30263 raft_consensus.cc:3037] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.434784 30410 raft_consensus.cc:3037] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:31.435675 30410 raft_consensus.cc:2454] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 1.
I20260806 02:07:31.435956 30263 raft_consensus.cc:2454] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0391128c25894e06bc0604df4da36b59 in term 1.
I20260806 02:07:31.435979 30046 leader_election.cc:302] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0391128c25894e06bc0604df4da36b59, 53466dff056243aaaf5b9aad8c4f90af; no voters:
I20260806 02:07:31.436141 30495 raft_consensus.cc:2781] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:31.436249 30495 raft_consensus.cc:686] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [term 1 LEADER]: Becoming Leader. State: Replica: 0391128c25894e06bc0604df4da36b59, State: Running, Role: LEADER
I20260806 02:07:31.436446 30495 consensus_queue.cc:237] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } }
I20260806 02:07:31.437083 30495 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: TxnStatusTablet state changed. Reason: New leader 0391128c25894e06bc0604df4da36b59. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.437182 30495 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:31.437515 30528 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:07:31.437729 30528 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:07:31.438493 29953 catalog_manager.cc:5495] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0391128c25894e06bc0604df4da36b59 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:31.477149 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.473656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.28.21.1:36345)
0806 02:07:31.473790 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.473795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.473820 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:07:31.473912 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.473914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.473959 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.474022 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:31.474282 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.474856 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.476072 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.476282 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.476354 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.476666 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.476735 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.476785 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.476973 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.477032 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:07:31.477152 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.473305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36345 (local address 127.0.0.1:52982)
0806 02:07:31.473675 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.473692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.473796 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.474069 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.474072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.474083 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:31.474199 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.474205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.474960 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.474964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.475946 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.475957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.476382 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.476386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.476479 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.476532 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.476820 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.476834 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.476906 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.477000 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260806 02:07:31.480297 30263 raft_consensus.cc:1261] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Refusing update from remote peer 0391128c25894e06bc0604df4da36b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.480309 30410 raft_consensus.cc:1261] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Refusing update from remote peer 0391128c25894e06bc0604df4da36b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.480785 30495 consensus_queue.cc:1035] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.481024 30490 consensus_queue.cc:1035] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.482550 30492 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: TxnStatusTablet state changed. Reason: New leader 0391128c25894e06bc0604df4da36b59. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.482668 30492 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.482825 30491 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: TxnStatusTablet state changed. Reason: New leader 0391128c25894e06bc0604df4da36b59. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.482937 30491 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.484588 30490 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.484691 30490 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:31.484799 30490 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.484884 30490 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 0391128c25894e06bc0604df4da36b59: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:31.485670 30501 mvcc.cc:205] Tried to move back new op lower bound from 7315382482858733568 to 7315382482685014016. Current Snapshot: MvccSnapshot[applied={T|T < 7315382482858733568}]
I20260806 02:07:31.485848 30492 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.485936 30492 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P ded7300cb8ba456081626002bcbb5ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.488447 30491 tablet_replica.cc:439] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0391128c25894e06bc0604df4da36b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 } } peers { permanent_uuid: "ded7300cb8ba456081626002bcbb5ba3" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41891 } } peers { permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 } } }
I20260806 02:07:31.488560 30491 tablet_replica.cc:442] T 0ba5e8f04d424850b42d1c32dadb89e5 P 53466dff056243aaaf5b9aad8c4f90af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:31.494611 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.490726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.28.21.2:41891)
0806 02:07:31.490837 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.490841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.490884 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:07:31.491023 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.491026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.491084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.491198 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:31.491356 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.492153 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.493041 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.493383 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.493484 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.493698 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.494229 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.494343 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.494397 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.494480 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:07:31.505607 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.501821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:44353 (local address 127.28.21.1:36345)
0806 02:07:31.501973 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.501981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.502196 (+ 215us) server_negotiation.cc:407] Connection header received
0806 02:07:31.502389 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.502393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.502476 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.502557 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:31.502847 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.503633 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.504482 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.504827 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.504928 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.505188 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.505269 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.505339 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.505417 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.505470 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:31.505988 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.501722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36345 (local address 127.28.21.2:44353)
0806 02:07:31.502114 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.502126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.502244 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.502579 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.502581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.502588 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:31.502735 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.502740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.503742 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.503744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.504371 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.504379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.504943 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.504946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.505020 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.505062 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.505353 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.505358 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.505407 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.505489 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260806 02:07:31.513357 30410 raft_consensus.cc:1261] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Refusing update from remote peer ded7300cb8ba456081626002bcbb5ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.513679 30116 raft_consensus.cc:1261] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Refusing update from remote peer ded7300cb8ba456081626002bcbb5ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:31.513878 30515 consensus_queue.cc:1035] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53466dff056243aaaf5b9aad8c4f90af" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.514187 30492 consensus_queue.cc:1035] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0391128c25894e06bc0604df4da36b59" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:31.518934 30499 mvcc.cc:205] Tried to move back new op lower bound from 7315382482993881088 to 7315382481638297600. Current Snapshot: MvccSnapshot[applied={T|T < 7315382482993881088}]
I20260806 02:07:31.540669 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.536364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:41891 (local address 127.28.21.1:54607)
0806 02:07:31.536819 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:31.536834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:31.536939 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:31.537169 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:31.537171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:31.537179 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:31.537324 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.537330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.538346 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.538349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.539318 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:31.539330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.539818 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.539822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:31.539913 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.539968 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.540342 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.540350 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:07:31.540424 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:07:31.540512 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":111,"threads_started":1}
I20260806 02:07:31.540704 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.536488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:54607 (local address 127.28.21.2:41891)
0806 02:07:31.536617 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.536622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.536911 (+ 289us) server_negotiation.cc:407] Connection header received
0806 02:07:31.537030 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.537032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.537074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.537136 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:31.537481 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.538234 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.539480 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.539696 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.539787 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.540169 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:31.540246 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:31.540361 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.540456 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.540506 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:07:31.569532 29950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:31.569722 29950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:31.572135 29950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7070b20bfb6541b296ea89a687073b02: Sending DeleteTablet for 3 replicas of tablet eb579aa7bf8e40d78433f0c0ca7829a3
I20260806 02:07:31.572808 30389 tablet_service.cc:1514] Processing DeleteTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:45916
I20260806 02:07:31.573005 30508 tablet_replica.cc:330] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: stopping tablet replica
I20260806 02:07:31.573129 30508 raft_consensus.cc:2229] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:31.573243 30508 raft_consensus.cc:2258] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.574071 30508 ts_tablet_manager.cc:1905] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.574203 30243 tablet_service.cc:1514] Processing DeleteTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33446
I20260806 02:07:31.574394 30095 tablet_service.cc:1514] Processing DeleteTablet for tablet eb579aa7bf8e40d78433f0c0ca7829a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:31 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51458
I20260806 02:07:31.574508 30509 tablet_replica.cc:330] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: stopping tablet replica
I20260806 02:07:31.574601 30507 tablet_replica.cc:330] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: stopping tablet replica
I20260806 02:07:31.574638 30509 raft_consensus.cc:2229] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:31.574721 30507 raft_consensus.cc:2229] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:31.574839 30507 raft_consensus.cc:2258] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.574940 30509 raft_consensus.cc:2258] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:31.575877 30509 ts_tablet_manager.cc:1905] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.578056 30508 ts_tablet_manager.cc:1918] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:31.578130 30508 log.cc:1196] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/eb579aa7bf8e40d78433f0c0ca7829a3
I20260806 02:07:31.578433 30508 ts_tablet_manager.cc:1939] T eb579aa7bf8e40d78433f0c0ca7829a3 P 53466dff056243aaaf5b9aad8c4f90af: Deleting consensus metadata
I20260806 02:07:31.579250 29939 catalog_manager.cc:4841] TS 53466dff056243aaaf5b9aad8c4f90af (127.28.21.3:34689): tablet eb579aa7bf8e40d78433f0c0ca7829a3 (table test-table [id=3485c972d17c438db50a1611e2a82f2c]) successfully deleted
I20260806 02:07:31.579420 30509 ts_tablet_manager.cc:1918] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:31.579594 30509 log.cc:1196] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/eb579aa7bf8e40d78433f0c0ca7829a3
I20260806 02:07:31.580009 30509 ts_tablet_manager.cc:1939] T eb579aa7bf8e40d78433f0c0ca7829a3 P ded7300cb8ba456081626002bcbb5ba3: Deleting consensus metadata
I20260806 02:07:31.580003 30507 ts_tablet_manager.cc:1905] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:31.580752 29936 catalog_manager.cc:4841] TS ded7300cb8ba456081626002bcbb5ba3 (127.28.21.2:41891): tablet eb579aa7bf8e40d78433f0c0ca7829a3 (table test-table [id=3485c972d17c438db50a1611e2a82f2c]) successfully deleted
I20260806 02:07:31.582882 30507 ts_tablet_manager.cc:1918] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:31.582970 30507 log.cc:1196] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/eb579aa7bf8e40d78433f0c0ca7829a3
I20260806 02:07:31.583231 30507 ts_tablet_manager.cc:1939] T eb579aa7bf8e40d78433f0c0ca7829a3 P 0391128c25894e06bc0604df4da36b59: Deleting consensus metadata
I20260806 02:07:31.583876 29936 catalog_manager.cc:4841] TS 0391128c25894e06bc0604df4da36b59 (127.28.21.1:36345): tablet eb579aa7bf8e40d78433f0c0ca7829a3 (table test-table [id=3485c972d17c438db50a1611e2a82f2c]) successfully deleted
Aug 06 02:07:31 dist-test-slave-gp6n krb5kdc[29904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:31.593466 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.574898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.28.21.1:36345)
0806 02:07:31.575020 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.575027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.580053 (+ 5026us) server_negotiation.cc:407] Connection header received
0806 02:07:31.580246 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.580249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.580296 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.580349 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:31.581294 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.582203 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.583806 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.584180 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.584259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.588790 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:31.588802 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:31.588820 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:31.588851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:31.591298 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.591827 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.591835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.591839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.591941 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.592247 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.592251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.592253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.592593 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:31.592864 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.593088 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.593271 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 02:07:31.593917 30156 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52996
I20260806 02:07:31.607908 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:31.595740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.28.21.62:41325)
0806 02:07:31.595875 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:31.595880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:31.598735 (+ 2855us) server_negotiation.cc:407] Connection header received
0806 02:07:31.598854 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:31.598858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:31.598922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:31.599028 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:31.599866 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.600702 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.602001 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:31.602385 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:31.602500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:31.604246 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:31.604253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:31.604257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:31.604289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:31.606009 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.606527 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.606530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.606532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.606614 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:31.606895 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:31.606898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:31.606900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:31.607100 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:31.607209 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:31.607616 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 02:07:31.607745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 02:07:31.609238 29950 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52318
I20260806 02:07:31.611387 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30027
I20260806 02:07:31.625085 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30172
I20260806 02:07:31.632265 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30321
I20260806 02:07:31.637858 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 29918
2026-08-06T02:07:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3004 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:31 dist-test-slave-gp6n krb5kdc[30550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:31 dist-test-slave-gp6n krb5kdc[30550](info): set up 2 sockets
Aug 06 02:07:31 dist-test-slave-gp6n krb5kdc[30550](info): commencing operation
krb5kdc: starting...
W20260806 02:07:32.728988 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:32 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:32.857522 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42999
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:32.970796 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:32.971136 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:32.971230 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:32.975436 30564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:32.975523 30564 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:07:32.975558 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:32.975602 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:32.975644 30564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:32.975680 30564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:32.981076 30564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:42999
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:32.982213 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:32.983381 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:32.992254 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:32.992264 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:32.992264 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:32.992650 30564 server_base.cc:1034] running on GCE node
I20260806 02:07:32.993162 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:32.994287 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:32.995472 30564 hybrid_clock.cc:648] HybridClock initialized: now 1785982052995462 us; error 34 us; skew 500 ppm
Aug 06 02:07:32 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:32.998572 30564 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:32.999554 30564 webserver.cc:460] Webserver started at http://127.28.21.62:35987/ using document root <none> and password file <none>
I20260806 02:07:33.000157 30564 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:33.000259 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:33.000517 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:33.002815 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "a61d528c642f40338bb8c2b4353ec204"
format_stamp: "Formatted at 2026-08-06 02:07:32 on dist-test-slave-gp6n"
server_key: "1ac07272cbc6a58d1ffe96ecfae609ed"
server_key_iv: "4e442f6fee43beb3c90bb9d47e9490d0"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.003458 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "a61d528c642f40338bb8c2b4353ec204"
format_stamp: "Formatted at 2026-08-06 02:07:32 on dist-test-slave-gp6n"
server_key: "1ac07272cbc6a58d1ffe96ecfae609ed"
server_key_iv: "4e442f6fee43beb3c90bb9d47e9490d0"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.007565 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:07:33.010242 30579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.011639 30564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:33.011786 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "a61d528c642f40338bb8c2b4353ec204"
format_stamp: "Formatted at 2026-08-06 02:07:32 on dist-test-slave-gp6n"
server_key: "1ac07272cbc6a58d1ffe96ecfae609ed"
server_key_iv: "4e442f6fee43beb3c90bb9d47e9490d0"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.011909 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:33.036827 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:33.039932 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:33.040153 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:33.048631 30564 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:42999
I20260806 02:07:33.048650 30643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:42999 every 8 connection(s)
I20260806 02:07:33.049603 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:33.052603 30644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.054693 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30564
I20260806 02:07:33.054836 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:33.055147 28756 external_mini_cluster.cc:1428] Setting key 30ea5858e1ec8fa735d4bcc6d0cc23c7
I20260806 02:07:33.060130 30644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Bootstrap starting.
I20260806 02:07:33.063103 30644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.064141 30644 log.cc:824] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.066325 30644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: No bootstrap required, opened a new log
I20260806 02:07:33.070935 30644 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } }
I20260806 02:07:33.071210 30644 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.071259 30644 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61d528c642f40338bb8c2b4353ec204, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.071432 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.056770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.28.21.62:42999)
0806 02:07:33.057245 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.057261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.057299 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:07:33.058028 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.058046 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.058392 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.058762 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.059758 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.060847 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.062229 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.062761 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.062889 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.065951 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.065957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.065973 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.066004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.068143 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.068632 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.068638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.068644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.068783 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.069070 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.069073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.069075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.069612 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.069800 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.070096 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.070448 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260806 02:07:33.071735 30644 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } }
I20260806 02:07:33.071887 30644 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:33.071974 30644 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:33.072072 30644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:33.073187 30644 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } }
I20260806 02:07:33.073539 30644 leader_election.cc:302] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61d528c642f40338bb8c2b4353ec204; no voters:
I20260806 02:07:33.073865 30644 leader_election.cc:288] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:33.074216 30649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:33.074468 30649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [term 1 LEADER]: Becoming Leader. State: Replica: a61d528c642f40338bb8c2b4353ec204, State: Running, Role: LEADER
I20260806 02:07:33.074806 30649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } }
I20260806 02:07:33.075132 30644 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:33.076673 30651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61d528c642f40338bb8c2b4353ec204. Latest consensus state: current_term: 1 leader_uuid: "a61d528c642f40338bb8c2b4353ec204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } } }
I20260806 02:07:33.076719 30650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61d528c642f40338bb8c2b4353ec204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61d528c642f40338bb8c2b4353ec204" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42999 } } }
I20260806 02:07:33.076833 30650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:33.076815 30651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204 [sys.catalog]: This master's current role is: LEADER
W20260806 02:07:33.080050 30665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:33.080144 30665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:33.080229 30663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:33.080989 30663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:33.087267 30663 catalog_manager.cc:1347] Generated new cluster ID: 96edbf50b60e442294be8e4c498e4a9a
I20260806 02:07:33.087654 30663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:33.117010 30663 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:33.118330 30663 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:33.136839 30663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Generated new TSK 0
I20260806 02:07:33.137728 30663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:33.178102 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42999
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:33.298044 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:33.298309 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:33.298357 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:33.302335 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:33.302425 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:33.302467 30671 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:07:33.302528 30671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:33.302589 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:33.307930 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:33.308997 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:33.310096 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:33.319761 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.319790 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.319761 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:33.320134 30671 server_base.cc:1034] running on GCE node
I20260806 02:07:33.320501 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:33.321110 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:33.322268 30671 hybrid_clock.cc:648] HybridClock initialized: now 1785982053322251 us; error 29 us; skew 500 ppm
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:33.325086 30671 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:33.326045 30671 webserver.cc:460] Webserver started at http://127.28.21.1:45353/ using document root <none> and password file <none>
I20260806 02:07:33.326679 30671 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:33.326771 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:33.327005 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:33.328995 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "19bc2db80c68a8e963dddd32af3dc198"
server_key_iv: "6d2497a951db128a967d985f59835b9e"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.329581 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "19bc2db80c68a8e963dddd32af3dc198"
server_key_iv: "6d2497a951db128a967d985f59835b9e"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.333292 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:07:33.335834 30688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.337064 30671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:33.337198 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "19bc2db80c68a8e963dddd32af3dc198"
server_key_iv: "6d2497a951db128a967d985f59835b9e"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.337309 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:33.346678 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:33.349532 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:33.349743 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:33.351261 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:33.351334 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.351393 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:33.351439 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.362337 30671 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:44023
I20260806 02:07:33.362363 30808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:44023 every 8 connection(s)
I20260806 02:07:33.363359 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:33.365110 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.351927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:53235 (local address 127.28.21.62:42999)
0806 02:07:33.352092 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.352099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.353408 (+ 1309us) server_negotiation.cc:407] Connection header received
0806 02:07:33.354588 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.354591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.354652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.354736 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.355996 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.356574 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.357710 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.358003 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.358129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.361133 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.361137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.361140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.361191 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.362874 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.363464 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.363470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.363474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.363564 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.364021 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.364028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.364031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.364281 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.364391 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.364848 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.364984 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:07:33.365658 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30671
I20260806 02:07:33.365792 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:33.366055 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.352777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.1:53235)
0806 02:07:33.353229 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.353279 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.354277 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.354902 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.354909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.355280 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.355834 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.355849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.356702 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.356705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.357578 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.357587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.358178 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.358182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.358345 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.359058 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.359074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.360938 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.362990 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.362999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.363015 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.363348 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.363656 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.363662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.363665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.363880 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.364398 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.364404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.364635 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.364793 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.365048 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":144,"threads_started":1}
I20260806 02:07:33.366086 28756 external_mini_cluster.cc:1428] Setting key 33960792264282c349f7f7188517ebb2
I20260806 02:07:33.367609 30810 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42999
I20260806 02:07:33.367771 30810 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:33.368368 30810 heartbeater.cc:511] Master 127.28.21.62:42999 requested a full tablet report, sending...
I20260806 02:07:33.370005 30596 ts_manager.cc:194] Registered new tserver with Master: 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023)
I20260806 02:07:33.371323 30596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:53235
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:33.378715 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.369960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:43843 (local address 127.28.21.62:42999)
0806 02:07:33.370113 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.370120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.370247 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:07:33.370360 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.370365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.370448 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.370537 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.371230 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.371952 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.373108 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.373524 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.373643 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.375238 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.375241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.375243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.375265 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.376847 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.377467 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.377470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.377471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.377548 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.377873 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.377875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.377877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.378046 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.378124 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.378502 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.378598 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:07:33.378778 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.369702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.1:43843)
0806 02:07:33.370157 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.370172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.370275 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.370672 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.370677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.370926 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.371087 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.371097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.372099 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.372102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.372973 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.372984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.373792 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.373799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.373942 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.374485 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.374501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.375099 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.377039 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.377046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.377051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.377357 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.377650 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.377655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.377660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.377769 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.378147 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.378153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.378273 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.378397 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.378597 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:33.399009 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42999
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:33.510564 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:33.510826 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:33.510898 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:33.515074 30819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:33.515146 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:33.515208 30819 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:07:33.515259 30819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:33.515322 30819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:33.520820 30819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:33.521811 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:33.522892 30819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:33.532816 30824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.532857 30825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.532912 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:33.533075 30819 server_base.cc:1034] running on GCE node
I20260806 02:07:33.533488 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:33.534229 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:33.535436 30819 hybrid_clock.cc:648] HybridClock initialized: now 1785982053535438 us; error 50 us; skew 500 ppm
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:33.538275 30819 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:33.539182 30819 webserver.cc:460] Webserver started at http://127.28.21.2:46833/ using document root <none> and password file <none>
I20260806 02:07:33.539714 30819 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:33.539788 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:33.539987 30819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:33.542202 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "440ef057c5874fb590cd9c7c94fb0dac"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "f81b9477ec47e52568696ef53fc4c499"
server_key_iv: "c7ddc8852e4e7dd4f498369511a04f6f"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.542830 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "440ef057c5874fb590cd9c7c94fb0dac"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "f81b9477ec47e52568696ef53fc4c499"
server_key_iv: "c7ddc8852e4e7dd4f498369511a04f6f"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.546674 30819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:07:33.549085 30834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.550357 30819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:33.550534 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "440ef057c5874fb590cd9c7c94fb0dac"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "f81b9477ec47e52568696ef53fc4c499"
server_key_iv: "c7ddc8852e4e7dd4f498369511a04f6f"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.550627 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:33.559254 30819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:33.561985 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:33.562162 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:33.563531 30819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:33.563606 30819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.563674 30819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:33.563719 30819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.574672 30819 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:36917
I20260806 02:07:33.574752 30955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:36917 every 8 connection(s)
I20260806 02:07:33.575783 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:33.575994 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30819
I20260806 02:07:33.576100 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:33.576457 28756 external_mini_cluster.cc:1428] Setting key d231be5dc66dcf0f424344df15eeeeb3
I20260806 02:07:33.578629 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.564343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:49025 (local address 127.28.21.62:42999)
0806 02:07:33.564483 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.564487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.565464 (+ 977us) server_negotiation.cc:407] Connection header received
0806 02:07:33.566611 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.566616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.566682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.566749 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.568465 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.569080 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.570965 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.571188 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.571300 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.574343 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.574346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.574349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.574373 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.576018 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.576949 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.576953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.576956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.577049 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.577425 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.577429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.577431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.577649 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.577748 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.578354 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.578514 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:07:33.579722 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.564845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.2:49025)
0806 02:07:33.565319 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.565358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.566334 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.567225 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.567235 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.567688 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.568313 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.568328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.569209 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.569213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.570833 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.570846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.571350 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.571354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.571526 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.572649 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.572675 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.574171 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.576185 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.576191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.576207 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.576551 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.577137 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.577142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.577144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.577312 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.577764 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.577769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.578099 (+ 330us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.578309 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.578683 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260806 02:07:33.581177 30956 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42999
I20260806 02:07:33.581305 30956 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:33.581857 30956 heartbeater.cc:511] Master 127.28.21.62:42999 requested a full tablet report, sending...
I20260806 02:07:33.582978 30592 ts_manager.cc:194] Registered new tserver with Master: 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917)
I20260806 02:07:33.583777 30592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:49025
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:33.591293 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.583565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:50355 (local address 127.28.21.62:42999)
0806 02:07:33.583661 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.583665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.584583 (+ 918us) server_negotiation.cc:407] Connection header received
0806 02:07:33.584729 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.584733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.584780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.584863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.585279 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.585769 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.586632 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.586845 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.586960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.588018 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.588021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.588024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.588054 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.589516 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.589950 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.589953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.589955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.590034 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.590318 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.590322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.590324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.590614 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.590751 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.591016 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.591133 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:07:33.591298 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.583502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.2:50355)
0806 02:07:33.584497 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.584509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.584636 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.584880 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.584883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.585079 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.585169 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.585174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.585891 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.585895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.586508 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.586517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.586963 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.586967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.587061 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.587561 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.587572 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.587883 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.589633 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.589638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.589640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.589838 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.590126 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.590130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.590132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.590225 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.590741 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.590744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.590842 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.590954 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.591097 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:33.607900 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42999
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:33.720340 30965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:33.720614 30965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:33.720695 30965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:33.724577 30965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:33.724665 30965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:33.724737 30965 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:07:33.724779 30965 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:07:33.724849 30965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:33.730309 30965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:33.731412 30965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:33.732455 30965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:33.742640 30971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.742626 30970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:33.742620 30973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:33.743247 30965 server_base.cc:1034] running on GCE node
I20260806 02:07:33.743702 30965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:33.744405 30965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:33.745609 30965 hybrid_clock.cc:648] HybridClock initialized: now 1785982053745600 us; error 31 us; skew 500 ppm
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:33.748493 30965 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:33.749523 30965 webserver.cc:460] Webserver started at http://127.28.21.3:35247/ using document root <none> and password file <none>
I20260806 02:07:33.750113 30965 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:33.750206 30965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:33.750512 30965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:33.752614 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "c5b57ba571a14150a52b12e209bcd5a5"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "d67bd228c5139eb41ff9e3cf80d53226"
server_key_iv: "dccd8c84cda0662c3b03c5808d6598e4"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.753181 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "c5b57ba571a14150a52b12e209bcd5a5"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "d67bd228c5139eb41ff9e3cf80d53226"
server_key_iv: "dccd8c84cda0662c3b03c5808d6598e4"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.757134 30965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:33.759938 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.761330 30965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:07:33.761485 30965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "c5b57ba571a14150a52b12e209bcd5a5"
format_stamp: "Formatted at 2026-08-06 02:07:33 on dist-test-slave-gp6n"
server_key: "d67bd228c5139eb41ff9e3cf80d53226"
server_key_iv: "dccd8c84cda0662c3b03c5808d6598e4"
server_key_version: "encryptionkey@0"
I20260806 02:07:33.761610 30965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:33.784432 30965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:33.787750 30965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:33.787971 30965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:33.789578 30965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:33.789654 30965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:33.789731 30965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:33.789775 30965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.801365 30965 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:36781
I20260806 02:07:33.801733 31103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:36781 every 8 connection(s)
I20260806 02:07:33.802312 30965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:33.804515 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.790530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:41269 (local address 127.28.21.62:42999)
0806 02:07:33.790706 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.790712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.791734 (+ 1022us) server_negotiation.cc:407] Connection header received
0806 02:07:33.793219 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.793222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.793292 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.793399 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.794856 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.795497 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.797010 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.797307 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.797443 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.800364 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.800368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.800371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.800410 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.802257 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.802897 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.802903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.802908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.803019 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.803412 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.803415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.803417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.803607 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.803688 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.804260 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.804391 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:07:33.805323 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 30965
I20260806 02:07:33.805387 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.791023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.3:41269)
0806 02:07:33.791544 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.791606 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.792955 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.793666 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.793673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.794098 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.794699 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.794714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.795640 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.795644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.796861 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.796871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.797504 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.797508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.797728 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.798483 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.798504 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.800160 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.802386 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.802393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.802427 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.802758 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.803138 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.803142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.803144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.803314 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.803726 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.803731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.803967 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.804130 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.804400 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260806 02:07:33.805469 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:33.805800 28756 external_mini_cluster.cc:1428] Setting key fc51f802ef39b49e35d3c9e5aaff180c
I20260806 02:07:33.806795 31105 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42999
I20260806 02:07:33.806963 31105 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:33.807557 31105 heartbeater.cc:511] Master 127.28.21.62:42999 requested a full tablet report, sending...
I20260806 02:07:33.808604 30596 ts_manager.cc:194] Registered new tserver with Master: c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:33.809226 30596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:41269
I20260806 02:07:33.816048 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.809163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.28.21.3:59295)
0806 02:07:33.809472 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.809491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.809592 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.809824 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.809827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.810034 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.810130 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.810136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.810947 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.810950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.811522 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.811531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.811982 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.811985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.812076 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.812538 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.812549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.812862 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.814435 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.814440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.814443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.814641 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.814949 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.814952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.814954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.815042 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.815458 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.815461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.815607 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.815726 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.815879 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":126,"threads_started":1}
I20260806 02:07:33.816037 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.809223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59295 (local address 127.28.21.62:42999)
0806 02:07:33.809376 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.809380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.809552 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:07:33.809669 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.809672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.809718 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.809803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.810249 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.810818 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.811645 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.811868 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.811977 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.813017 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.813022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.813030 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.813051 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.814297 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.814741 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.814745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.814747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.814834 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.815135 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.815140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.815141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.815356 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.815469 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.815778 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.815883 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:07:33.820209 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.846987 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.838683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.28.21.62:42999)
0806 02:07:33.838848 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.838851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.838864 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:33.838902 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.838905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.838943 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.839036 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.839754 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.840279 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.841191 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.841413 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.841522 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.843964 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.843968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.843970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.843994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.845244 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.845699 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.845703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.845705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.845795 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.846080 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.846083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.846085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.846265 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.846435 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.846693 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.846822 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:07:33.850024 30596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:33.853354 30596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:33.871354 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.863272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.0.0.1:36982)
0806 02:07:33.863843 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.863894 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.864046 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.866097 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.866100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.866118 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:33.866229 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.866235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.867940 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.867944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.869524 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.869538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.870786 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.870789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.870897 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.870947 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.871036 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.871170 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":126,"spinlock_wait_cycles":8960,"thread_start_us":89,"threads_started":1}
I20260806 02:07:33.871804 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.864122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.28.21.2:36917)
0806 02:07:33.864499 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.864514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.864545 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:07:33.865155 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.865162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.865457 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.865606 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:33.866234 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.867738 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.869821 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.870643 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.870739 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.870930 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.871273 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.871376 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260806 02:07:33.872026 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.864525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.28.21.1:44023)
0806 02:07:33.864862 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.864881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.864906 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:07:33.864987 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.864994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.865940 (+ 946us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.866526 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:33.866675 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.867790 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.869757 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.870643 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.871575 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.871720 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.871769 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.871872 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
I20260806 02:07:33.872565 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.863476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36917 (local address 127.0.0.1:48624)
0806 02:07:33.863853 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.863897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.864969 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.865833 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.865841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.865879 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:33.866104 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.866112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.867910 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.867913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.869668 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.869683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.871029 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.871033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.871121 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.871141 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.872349 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.872427 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":88,"spinlock_wait_cycles":3840,"thread_start_us":103,"threads_started":1}
I20260806 02:07:33.872838 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.864543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.0.0.1:51346)
0806 02:07:33.865395 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.865410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.866386 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.867100 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.867103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.867124 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:33.867239 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.867303 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.869606 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.869609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.870703 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.870715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.872160 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.872164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.872304 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.872340 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.872424 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.872512 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":106,"threads_started":1}
I20260806 02:07:33.873656 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.864977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.28.21.3:36781)
0806 02:07:33.865765 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.865777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.865801 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:07:33.866587 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.866593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.866796 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.866950 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:33.867444 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.868703 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.870854 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.871792 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.871910 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.872086 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.873414 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.873514 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":116,"threads_started":1}
I20260806 02:07:33.875130 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 (DEFAULT_TABLE table=test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:33.875909 30888 tablet_service.cc:1467] Processing CreateTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 (DEFAULT_TABLE table=test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:33.876366 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f5ee53e4a749049ca2ef6865aeb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.876986 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f5ee53e4a749049ca2ef6865aeb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.877664 31032 tablet_service.cc:1467] Processing CreateTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 (DEFAULT_TABLE table=test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:33.879575 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f5ee53e4a749049ca2ef6865aeb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.884181 31128 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Bootstrap starting.
I20260806 02:07:33.885953 31129 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Bootstrap starting.
I20260806 02:07:33.886641 31128 tablet_bootstrap.cc:654] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.887377 31128 log.cc:824] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:33.888119 31129 tablet_bootstrap.cc:654] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.888160 31132 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Bootstrap starting.
I20260806 02:07:33.888962 31129 log.cc:824] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:33.889423 31128 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: No bootstrap required, opened a new log
I20260806 02:07:33.889600 31128 ts_tablet_manager.cc:1397] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:07:33.890184 31132 tablet_bootstrap.cc:654] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.890839 31129 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: No bootstrap required, opened a new log
I20260806 02:07:33.890977 31132 log.cc:824] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:33.891009 31129 ts_tablet_manager.cc:1397] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:33.894097 31132 tablet_bootstrap.cc:492] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: No bootstrap required, opened a new log
I20260806 02:07:33.894129 31128 raft_consensus.cc:348] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.894426 31128 raft_consensus.cc:374] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.894480 31128 raft_consensus.cc:729] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.894330 31132 ts_tablet_manager.cc:1397] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 02:07:33.894977 31128 consensus_queue.cc:260] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.895246 31129 raft_consensus.cc:348] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.895522 31129 raft_consensus.cc:374] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.895596 31129 raft_consensus.cc:729] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.895926 31128 ts_tablet_manager.cc:1428] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:07:33.896301 30810 heartbeater.cc:503] Master 127.28.21.62:42999 was elected leader, sending a full tablet report...
I20260806 02:07:33.897110 31129 consensus_queue.cc:260] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.898068 31129 ts_tablet_manager.cc:1428] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:07:33.898140 31132 raft_consensus.cc:348] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.898367 31132 raft_consensus.cc:374] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.898444 31132 raft_consensus.cc:729] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.898460 30956 heartbeater.cc:503] Master 127.28.21.62:42999 was elected leader, sending a full tablet report...
I20260806 02:07:33.898841 31132 consensus_queue.cc:260] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.899482 31105 heartbeater.cc:503] Master 127.28.21.62:42999 was elected leader, sending a full tablet report...
I20260806 02:07:33.900754 31132 ts_tablet_manager.cc:1428] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:07:33.911082 31138 raft_consensus.cc:482] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:33.911276 31138 raft_consensus.cc:504] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.912698 31138 leader_election.cc:288] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917)
I20260806 02:07:33.917855 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.912978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:52197 (local address 127.28.21.2:36917)
0806 02:07:33.913105 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.913110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.913124 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:07:33.913209 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.913212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.913261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.913338 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:33.913707 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.914703 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.916532 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.917146 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.917265 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.917327 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.917626 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.917728 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:07:33.917927 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.912888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36917 (local address 127.28.21.3:52197)
0806 02:07:33.913008 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.913024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.913117 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.913398 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.913402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.913423 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:33.913559 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.913565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.914872 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.914878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.916374 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.916391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.917293 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.917299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.917448 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.917505 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.917596 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.917708 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":6,"spinlock_wait_cycles":3456}
I20260806 02:07:33.917948 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.913298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:60265 (local address 127.28.21.1:44023)
0806 02:07:33.913434 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.913439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.913451 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:07:33.913502 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.913504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.913538 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.913598 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:33.914476 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.915194 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.916554 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.917179 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.917258 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.917325 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.917737 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.917826 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:07:33.918325 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.912888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.28.21.3:60265)
0806 02:07:33.913308 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.913324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.913415 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.913635 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.913640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.913659 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:33.914317 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.914326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.915338 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.915341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.916400 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.916413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.917449 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.917453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.917584 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.917611 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.918086 (+ 475us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.918189 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":193,"thread_start_us":134,"threads_started":1}
I20260806 02:07:33.918630 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f5ee53e4a749049ca2ef6865aeb307" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac" is_pre_election: true
I20260806 02:07:33.918931 30910 raft_consensus.cc:2454] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 0.
I20260806 02:07:33.919011 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f5ee53e4a749049ca2ef6865aeb307" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" is_pre_election: true
I20260806 02:07:33.919306 30761 raft_consensus.cc:2454] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 0.
I20260806 02:07:33.919420 30984 leader_election.cc:302] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440ef057c5874fb590cd9c7c94fb0dac, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:33.919766 31138 raft_consensus.cc:2781] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:33.919909 31138 raft_consensus.cc:482] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:33.919979 31138 raft_consensus.cc:3037] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:33.921446 31138 raft_consensus.cc:504] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.921849 31138 leader_election.cc:288] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917)
I20260806 02:07:33.922171 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f5ee53e4a749049ca2ef6865aeb307" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac"
I20260806 02:07:33.922194 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f5ee53e4a749049ca2ef6865aeb307" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
I20260806 02:07:33.922338 30761 raft_consensus.cc:3037] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:33.922319 30910 raft_consensus.cc:3037] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:33.923650 30910 raft_consensus.cc:2454] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 1.
I20260806 02:07:33.923659 30761 raft_consensus.cc:2454] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 1.
I20260806 02:07:33.923954 30984 leader_election.cc:302] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440ef057c5874fb590cd9c7c94fb0dac, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:33.924194 31138 raft_consensus.cc:2781] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:33.924437 31138 raft_consensus.cc:686] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Running, Role: LEADER
I20260806 02:07:33.924790 31138 consensus_queue.cc:237] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.928413 30592 catalog_manager.cc:5495] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } health_report { overall_health: HEALTHY } } }
Aug 06 02:07:33 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:33.972752 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.964308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.28.21.62:42999)
0806 02:07:33.964451 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:33.964455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:33.964612 (+ 157us) server_negotiation.cc:407] Connection header received
0806 02:07:33.964695 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:33.964698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:33.964738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:33.964808 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:33.965550 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.966138 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.967026 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.967236 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.967318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.969675 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:33.969678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:33.969680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:33.969701 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:33.971237 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.971653 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.971656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.971657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.971725 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:33.971963 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:33.971965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:33.971966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:33.972120 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:33.972192 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:33.972495 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:07:33.972616 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":60}
I20260806 02:07:33.972754 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:33.964206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42999 (local address 127.0.0.1:45264)
0806 02:07:33.964506 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:33.964519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:33.964622 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:33.964845 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:33.964848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:33.965260 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:33.965403 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.965410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.966236 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.966238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.966925 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:33.966934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:33.967333 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:33.967335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:33.967419 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:33.968030 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:33.968042 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:33.969488 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:33.971334 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.971348 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.971352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.971569 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.971804 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:33.971806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:33.971808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:33.971888 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:33.972208 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:33.972214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:33.972313 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 02:07:33.972437 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:07:33.972563 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
I20260806 02:07:33.976828 30592 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:45264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:07:33.977628 30592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:33.983101 30888 tablet_service.cc:1467] Processing CreateTablet for tablet df1006714fd4480b94a3a616001f1761 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=684de51bf70c423db8fa7ac7800a07a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:33.982931 30741 tablet_service.cc:1467] Processing CreateTablet for tablet df1006714fd4480b94a3a616001f1761 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=684de51bf70c423db8fa7ac7800a07a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:33.983043 31032 tablet_service.cc:1467] Processing CreateTablet for tablet df1006714fd4480b94a3a616001f1761 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=684de51bf70c423db8fa7ac7800a07a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:07:33.983753 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1006714fd4480b94a3a616001f1761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.983753 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1006714fd4480b94a3a616001f1761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.983757 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1006714fd4480b94a3a616001f1761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:33.986500 31132 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: Bootstrap starting.
I20260806 02:07:33.986617 31129 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: Bootstrap starting.
I20260806 02:07:33.986617 31128 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: Bootstrap starting.
I20260806 02:07:33.987509 31132 tablet_bootstrap.cc:654] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.987658 31128 tablet_bootstrap.cc:654] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.987658 31129 tablet_bootstrap.cc:654] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:33.989207 31128 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: No bootstrap required, opened a new log
I20260806 02:07:33.989275 31128 ts_tablet_manager.cc:1397] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:33.989327 31132 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: No bootstrap required, opened a new log
I20260806 02:07:33.989425 31132 ts_tablet_manager.cc:1397] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:33.989488 31129 tablet_bootstrap.cc:492] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: No bootstrap required, opened a new log
I20260806 02:07:33.989564 31129 ts_tablet_manager.cc:1397] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:33.989734 31128 raft_consensus.cc:348] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.989850 31128 raft_consensus.cc:374] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.989890 31128 raft_consensus.cc:729] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.989866 31132 raft_consensus.cc:348] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.989970 31132 raft_consensus.cc:374] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.990020 31132 raft_consensus.cc:729] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.989985 31129 raft_consensus.cc:348] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.990089 31129 raft_consensus.cc:374] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:33.990135 31129 raft_consensus.cc:729] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Initialized, Role: FOLLOWER
I20260806 02:07:33.990075 31128 consensus_queue.cc:260] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.990152 31132 consensus_queue.cc:260] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.990255 31129 consensus_queue.cc:260] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:33.990427 31128 ts_tablet_manager.cc:1428] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:07:33.990475 31132 ts_tablet_manager.cc:1428] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:33.990376 31136 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:33.990518 31129 ts_tablet_manager.cc:1428] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:33.990559 31136 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:33.990670 31138 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:33.990789 31138 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:33.990784 31137 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:33.990932 31137 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 02:07:34.053494 31106 tablet.cc:2369] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:07:34.077443 30957 tablet.cc:2369] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:34.091473 31138 raft_consensus.cc:482] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:34.091651 31138 raft_consensus.cc:504] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.092098 31138 leader_election.cc:288] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917)
I20260806 02:07:34.092636 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1006714fd4480b94a3a616001f1761" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" is_pre_election: true
I20260806 02:07:34.092648 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1006714fd4480b94a3a616001f1761" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac" is_pre_election: true
I20260806 02:07:34.092777 30761 raft_consensus.cc:2454] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 0.
I20260806 02:07:34.092806 30910 raft_consensus.cc:2454] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 0.
I20260806 02:07:34.093144 30984 leader_election.cc:302] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440ef057c5874fb590cd9c7c94fb0dac, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:34.093334 31138 raft_consensus.cc:2781] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:34.093417 31138 raft_consensus.cc:482] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:34.093462 31138 raft_consensus.cc:3037] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.094672 31138 raft_consensus.cc:504] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.094988 31138 leader_election.cc:288] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917)
I20260806 02:07:34.095321 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1006714fd4480b94a3a616001f1761" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac"
I20260806 02:07:34.095376 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1006714fd4480b94a3a616001f1761" candidate_uuid: "c5b57ba571a14150a52b12e209bcd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
I20260806 02:07:34.095465 30910 raft_consensus.cc:3037] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.095497 30761 raft_consensus.cc:3037] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.096400 30910 raft_consensus.cc:2454] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 1.
I20260806 02:07:34.096525 30761 raft_consensus.cc:2454] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5b57ba571a14150a52b12e209bcd5a5 in term 1.
I20260806 02:07:34.096697 30984 leader_election.cc:302] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440ef057c5874fb590cd9c7c94fb0dac, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:34.096875 31138 raft_consensus.cc:2781] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:34.096978 31138 raft_consensus.cc:686] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Running, Role: LEADER
I20260806 02:07:34.097136 31138 consensus_queue.cc:237] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.097893 31140 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: TxnStatusTablet state changed. Reason: New leader c5b57ba571a14150a52b12e209bcd5a5. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.098013 31140 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:34.098515 31152 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:07:34.098788 31152 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:07:34.099053 30596 catalog_manager.cc:5495] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } health_report { overall_health: HEALTHY } } }
I20260806 02:07:34.113344 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.109040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.0.0.1:51350)
0806 02:07:34.109193 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.109216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.109311 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.109667 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.109669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.109674 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:34.109764 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.109769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.110818 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.110822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.111526 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.111535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.112139 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.112143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.112224 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.112272 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.113050 (+ 778us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.113067 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.113148 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.113226 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 02:07:34.113373 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.109154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.28.21.3:36781)
0806 02:07:34.109266 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.109270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.109301 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:07:34.109455 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.109458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.109538 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.109643 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.109866 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.110669 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.111655 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.112005 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.112117 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.112426 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.112930 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.113046 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.113182 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.113244 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 02:07:34.115197 30811 tablet.cc:2369] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:34.119370 30761 raft_consensus.cc:1261] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Refusing update from remote peer c5b57ba571a14150a52b12e209bcd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.119463 30910 raft_consensus.cc:1261] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Refusing update from remote peer c5b57ba571a14150a52b12e209bcd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.120065 31138 consensus_queue.cc:1035] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.120253 31140 consensus_queue.cc:1035] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.124459 31137 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: TxnStatusTablet state changed. Reason: New leader c5b57ba571a14150a52b12e209bcd5a5. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.124594 31137 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:34.124954 31136 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: TxnStatusTablet state changed. Reason: New leader c5b57ba571a14150a52b12e209bcd5a5. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.125094 31136 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:34.127017 31138 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.127125 31138 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:34.127800 31138 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.127915 31138 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P c5b57ba571a14150a52b12e209bcd5a5: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:07:34.127979 31155 mvcc.cc:205] Tried to move back new op lower bound from 7315382493665726464 to 7315382493583409152. Current Snapshot: MvccSnapshot[applied={T|T < 7315382493665726464}]
I20260806 02:07:34.129231 31136 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.129387 31136 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 2ccd39b6bfe4437d924d8f6cd939d835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:34.131369 31137 tablet_replica.cc:439] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5b57ba571a14150a52b12e209bcd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } } }
I20260806 02:07:34.131491 31137 tablet_replica.cc:442] T df1006714fd4480b94a3a616001f1761 P 440ef057c5874fb590cd9c7c94fb0dac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:07:34.133092 31157 mvcc.cc:205] Tried to move back new op lower bound from 7315382493665726464 to 7315382493583409152. Current Snapshot: MvccSnapshot[applied={T|T < 7315382493665726464}]
I20260806 02:07:34.142292 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.138579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.28.21.3:36781)
0806 02:07:34.138719 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.138722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.138735 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:34.138777 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.138779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.138825 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.138933 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.139180 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.139749 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.140991 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.141361 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.141451 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.141777 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.141865 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.141951 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.142059 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.142110 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:07:34.154044 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.150026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.28.21.3:59507)
0806 02:07:34.150191 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.150204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.150278 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.150753 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.150756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.150763 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:34.150901 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.150906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.151859 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.151862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.152539 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.152549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.153132 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.153135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.153220 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.153303 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.153646 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.153652 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.153714 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.153786 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":17}
I20260806 02:07:34.154044 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.150386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59507 (local address 127.28.21.3:36781)
0806 02:07:34.150549 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.150554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.150567 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:34.150607 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.150610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.150652 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.150726 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.151005 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.151736 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.152654 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.152993 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.153097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.153480 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.153556 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.153625 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.153743 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.153792 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":99}
I20260806 02:07:34.161934 30910 raft_consensus.cc:1261] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Refusing update from remote peer c5b57ba571a14150a52b12e209bcd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.162024 30761 raft_consensus.cc:1261] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Refusing update from remote peer c5b57ba571a14150a52b12e209bcd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.162442 31140 consensus_queue.cc:1035] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.162637 31138 consensus_queue.cc:1035] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.168659 31156 mvcc.cc:205] Tried to move back new op lower bound from 7315382493843738624 to 7315382492878999552. Current Snapshot: MvccSnapshot[applied={T|T < 7315382493843738624}]
I20260806 02:07:34.201614 30592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:34.201854 30592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:34.204414 30592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Sending DeleteTablet for 3 replicas of tablet 30f5ee53e4a749049ca2ef6865aeb307
I20260806 02:07:34.205116 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:34 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:36982
I20260806 02:07:34.205144 30888 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:34 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:48624
I20260806 02:07:34.205673 31171 tablet_replica.cc:330] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: stopping tablet replica
I20260806 02:07:34.205672 31170 tablet_replica.cc:330] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: stopping tablet replica
I20260806 02:07:34.205928 31031 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f5ee53e4a749049ca2ef6865aeb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:34 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51346
I20260806 02:07:34.206008 31171 raft_consensus.cc:2229] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:34.206008 31170 raft_consensus.cc:2229] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:34.206243 31171 raft_consensus.cc:2258] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:34.206458 31170 raft_consensus.cc:2258] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:34.207432 30592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:07:34.207988 31170 ts_tablet_manager.cc:1905] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:07:34.207998 30592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:34.208159 31171 ts_tablet_manager.cc:1905] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:34.211197 31170 ts_tablet_manager.cc:1918] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:34.211243 31171 ts_tablet_manager.cc:1918] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:34.211293 31170 log.cc:1196] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/30f5ee53e4a749049ca2ef6865aeb307
I20260806 02:07:34.211326 31171 log.cc:1196] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/30f5ee53e4a749049ca2ef6865aeb307
I20260806 02:07:34.211635 31171 ts_tablet_manager.cc:1939] T 30f5ee53e4a749049ca2ef6865aeb307 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting consensus metadata
I20260806 02:07:34.211637 31170 ts_tablet_manager.cc:1939] T 30f5ee53e4a749049ca2ef6865aeb307 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting consensus metadata
I20260806 02:07:34.211964 31172 tablet_replica.cc:330] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: stopping tablet replica
I20260806 02:07:34.212347 31172 raft_consensus.cc:2229] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:34.212754 31172 raft_consensus.cc:2258] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:34.212909 30583 catalog_manager.cc:4841] TS 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917): tablet 30f5ee53e4a749049ca2ef6865aeb307 (table test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]) successfully deleted
I20260806 02:07:34.214277 30741 tablet_service.cc:1467] Processing CreateTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 (DEFAULT_TABLE table=test-table [id=3315d4be48a34f77b8d142eb6141d447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:34.214460 31031 tablet_service.cc:1467] Processing CreateTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 (DEFAULT_TABLE table=test-table [id=3315d4be48a34f77b8d142eb6141d447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:34.214587 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abee7ba63b6349c2a240dfccbaf277c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:34.214576 30888 tablet_service.cc:1467] Processing CreateTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 (DEFAULT_TABLE table=test-table [id=3315d4be48a34f77b8d142eb6141d447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:34.214713 31031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abee7ba63b6349c2a240dfccbaf277c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:34.214820 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abee7ba63b6349c2a240dfccbaf277c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:34.217476 31132 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Bootstrap starting.
I20260806 02:07:34.218601 31132 tablet_bootstrap.cc:654] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:34.219242 31129 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Bootstrap starting.
I20260806 02:07:34.220439 31129 tablet_bootstrap.cc:654] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:34.221757 30580 catalog_manager.cc:4841] TS 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023): tablet 30f5ee53e4a749049ca2ef6865aeb307 (table test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]) successfully deleted
I20260806 02:07:34.221865 31132 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: No bootstrap required, opened a new log
I20260806 02:07:34.221951 31132 ts_tablet_manager.cc:1397] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:07:34.222525 31132 raft_consensus.cc:348] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.222666 31132 raft_consensus.cc:374] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:34.222703 31132 raft_consensus.cc:729] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Initialized, Role: FOLLOWER
I20260806 02:07:34.223227 31129 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: No bootstrap required, opened a new log
I20260806 02:07:34.223330 31129 ts_tablet_manager.cc:1397] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:07:34.223843 31129 raft_consensus.cc:348] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.223966 31129 raft_consensus.cc:374] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:34.224009 31129 raft_consensus.cc:729] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Initialized, Role: FOLLOWER
I20260806 02:07:34.224222 31128 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Bootstrap starting.
I20260806 02:07:34.224211 31132 consensus_queue.cc:260] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.224462 31129 consensus_queue.cc:260] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.224779 31129 ts_tablet_manager.cc:1428] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:34.225242 31128 tablet_bootstrap.cc:654] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:34.225958 31132 ts_tablet_manager.cc:1428] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:07:34.227192 31172 ts_tablet_manager.cc:1905] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:34.230154 31172 ts_tablet_manager.cc:1918] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:34.230294 31172 log.cc:1196] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/30f5ee53e4a749049ca2ef6865aeb307
I20260806 02:07:34.230731 31172 ts_tablet_manager.cc:1939] T 30f5ee53e4a749049ca2ef6865aeb307 P c5b57ba571a14150a52b12e209bcd5a5: Deleting consensus metadata
I20260806 02:07:34.231348 31128 tablet_bootstrap.cc:492] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: No bootstrap required, opened a new log
I20260806 02:07:34.231436 31128 ts_tablet_manager.cc:1397] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 02:07:34.231910 31128 raft_consensus.cc:348] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.232028 31128 raft_consensus.cc:374] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:34.232059 31128 raft_consensus.cc:729] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Initialized, Role: FOLLOWER
I20260806 02:07:34.231990 30582 catalog_manager.cc:4841] TS c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781): tablet 30f5ee53e4a749049ca2ef6865aeb307 (table test-table [id=8c38bd1ce8f445bc816eba416fb6bcd8]) successfully deleted
I20260806 02:07:34.232177 31128 consensus_queue.cc:260] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.232424 31128 ts_tablet_manager.cc:1428] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:34.270304 31136 raft_consensus.cc:482] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:34.270483 31136 raft_consensus.cc:504] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.271684 31136 leader_election.cc:288] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:34.276785 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.272045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:37979 (local address 127.28.21.3:36781)
0806 02:07:34.272161 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.272165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.272346 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:07:34.272526 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.272529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.272577 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.272649 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:34.273027 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.273630 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.275387 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.275918 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.276006 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.276082 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.276609 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.276682 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:07:34.276813 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.271918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36917 (local address 127.28.21.1:55025)
0806 02:07:34.272662 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.272675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.272810 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.273059 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.273062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.273079 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:34.273182 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.273187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.274206 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.274209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.275377 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.275385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.276292 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.276298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.276436 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.276486 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.276561 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.276634 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":402,"thread_start_us":180,"threads_started":1}
I20260806 02:07:34.276790 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.272102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:55025 (local address 127.28.21.2:36917)
0806 02:07:34.272240 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.272245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.272749 (+ 504us) server_negotiation.cc:407] Connection header received
0806 02:07:34.272930 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.272934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.272981 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.273048 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:34.273285 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.274079 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.275496 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.276148 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.276259 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.276340 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.276597 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.276671 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:07:34.277196 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.271918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.28.21.1:37979)
0806 02:07:34.272248 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.272264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.272395 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.272677 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.272682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.272705 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:34.272859 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.272868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.273740 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.273744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.275238 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.275249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.276193 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.276195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.276396 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.276481 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.276993 (+ 512us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.277069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":40,"thread_start_us":74,"threads_started":1}
I20260806 02:07:34.277366 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abee7ba63b6349c2a240dfccbaf277c4" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac" is_pre_election: true
I20260806 02:07:34.277508 30910 raft_consensus.cc:2454] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 0.
I20260806 02:07:34.277685 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abee7ba63b6349c2a240dfccbaf277c4" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5" is_pre_election: true
I20260806 02:07:34.277889 30692 leader_election.cc:302] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccd39b6bfe4437d924d8f6cd939d835, 440ef057c5874fb590cd9c7c94fb0dac; no voters:
I20260806 02:07:34.277927 31054 raft_consensus.cc:2454] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 0.
I20260806 02:07:34.278235 31136 raft_consensus.cc:2781] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:34.278304 31136 raft_consensus.cc:482] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:34.278342 31136 raft_consensus.cc:3037] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.279567 31136 raft_consensus.cc:504] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.279946 31136 leader_election.cc:288] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 election: Requested vote from peers 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:34.280263 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abee7ba63b6349c2a240dfccbaf277c4" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5"
I20260806 02:07:34.280237 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abee7ba63b6349c2a240dfccbaf277c4" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac"
I20260806 02:07:34.280388 31054 raft_consensus.cc:3037] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.280390 30910 raft_consensus.cc:3037] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:34.281423 31054 raft_consensus.cc:2454] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 1.
I20260806 02:07:34.281446 30910 raft_consensus.cc:2454] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 1.
I20260806 02:07:34.281705 30691 leader_election.cc:302] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccd39b6bfe4437d924d8f6cd939d835, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:34.282002 31136 raft_consensus.cc:2781] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:34.282219 31136 raft_consensus.cc:686] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Running, Role: LEADER
I20260806 02:07:34.282492 31136 consensus_queue.cc:237] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:34.284900 30592 catalog_manager.cc:5495] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:34.307997 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.303740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.28.21.1:44023)
0806 02:07:34.304034 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.304038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.304047 (+ 9us) server_negotiation.cc:407] Connection header received
0806 02:07:34.304117 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.304119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.304166 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.304246 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.304541 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.305108 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.306295 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.306596 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.306704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.307007 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.307546 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.307637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.307760 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.307817 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260806 02:07:34.318682 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.315230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:39519 (local address 127.28.21.3:36781)
0806 02:07:34.315369 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.315375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.315556 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:07:34.315705 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.315708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.315782 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.315865 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.316108 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.316685 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.317551 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.317781 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.317904 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.318195 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.318270 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.318349 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.318484 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.318538 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:07:34.318838 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.315120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.28.21.1:39519)
0806 02:07:34.315472 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.315485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.315601 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.315874 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.315877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.315884 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:34.316020 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.316027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.316791 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.316796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.317430 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.317438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.317906 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.317909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.318012 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.318052 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.318360 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.318369 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.318468 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.318653 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260806 02:07:34.325563 30910 raft_consensus.cc:1261] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Refusing update from remote peer 2ccd39b6bfe4437d924d8f6cd939d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.325637 31054 raft_consensus.cc:1261] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 2ccd39b6bfe4437d924d8f6cd939d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:34.326076 31136 consensus_queue.cc:1035] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.326283 31178 consensus_queue.cc:1035] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:34.345170 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.341818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:56909 (local address 127.28.21.1:44023)
0806 02:07:34.341948 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:34.341953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:34.341968 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:07:34.342113 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:34.342116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:34.342164 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:34.342224 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:34.342584 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.343136 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.344245 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.344476 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.344558 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.344807 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.344873 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.344920 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:34.345029 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 02:07:34.345070 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:07:34.345235 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:34.341744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.28.21.3:56909)
0806 02:07:34.341908 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:34.341922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:34.341995 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:34.342338 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:34.342341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:34.342349 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:34.342462 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.342468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.343234 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.343237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.344141 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:34.344151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:34.344584 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:34.344588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:34.344677 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:34.344708 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:34.344947 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:34.344953 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:07:34.345023 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:07:34.345102 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 02:07:34.346228 31156 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:07:35.912099 30595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:35.912247 30595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:35.914727 30595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Sending DeleteTablet for 3 replicas of tablet abee7ba63b6349c2a240dfccbaf277c4
I20260806 02:07:35.915330 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:35 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:36982
I20260806 02:07:35.915535 30888 tablet_service.cc:1514] Processing DeleteTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:35 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:48624
I20260806 02:07:35.915731 31032 tablet_service.cc:1514] Processing DeleteTablet for tablet abee7ba63b6349c2a240dfccbaf277c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:35 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51346
I20260806 02:07:35.916033 31215 tablet_replica.cc:330] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: stopping tablet replica
I20260806 02:07:35.916137 31214 tablet_replica.cc:330] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: stopping tablet replica
I20260806 02:07:35.916244 31215 raft_consensus.cc:2229] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:35.916397 31214 raft_consensus.cc:2229] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:35.916513 31215 raft_consensus.cc:2258] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:35.916719 31214 raft_consensus.cc:2258] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:35.917331 31215 ts_tablet_manager.cc:1905] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:35.917593 31214 ts_tablet_manager.cc:1905] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:35.918586 31216 tablet_replica.cc:330] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: stopping tablet replica
I20260806 02:07:35.918776 31216 raft_consensus.cc:2229] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:35.918952 31216 raft_consensus.cc:2258] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:35.919673 31216 ts_tablet_manager.cc:1905] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:35.920396 31215 ts_tablet_manager.cc:1918] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:07:35.920495 31215 log.cc:1196] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/abee7ba63b6349c2a240dfccbaf277c4
I20260806 02:07:35.920759 31215 ts_tablet_manager.cc:1939] T abee7ba63b6349c2a240dfccbaf277c4 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting consensus metadata
I20260806 02:07:35.920791 31214 ts_tablet_manager.cc:1918] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:07:35.920914 31214 log.cc:1196] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/abee7ba63b6349c2a240dfccbaf277c4
I20260806 02:07:35.921334 31214 ts_tablet_manager.cc:1939] T abee7ba63b6349c2a240dfccbaf277c4 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting consensus metadata
I20260806 02:07:35.921639 30583 catalog_manager.cc:4841] TS 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917): tablet abee7ba63b6349c2a240dfccbaf277c4 (table test-table [id=3315d4be48a34f77b8d142eb6141d447]) successfully deleted
I20260806 02:07:35.922073 30580 catalog_manager.cc:4841] TS 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023): tablet abee7ba63b6349c2a240dfccbaf277c4 (table test-table [id=3315d4be48a34f77b8d142eb6141d447]) successfully deleted
I20260806 02:07:35.922554 31216 ts_tablet_manager.cc:1918] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:07:35.922618 31216 log.cc:1196] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/abee7ba63b6349c2a240dfccbaf277c4
I20260806 02:07:35.922870 31216 ts_tablet_manager.cc:1939] T abee7ba63b6349c2a240dfccbaf277c4 P c5b57ba571a14150a52b12e209bcd5a5: Deleting consensus metadata
I20260806 02:07:35.923707 30582 catalog_manager.cc:4841] TS c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781): tablet abee7ba63b6349c2a240dfccbaf277c4 (table test-table [id=3315d4be48a34f77b8d142eb6141d447]) successfully deleted
Aug 06 02:07:35 dist-test-slave-gp6n krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:07:35 dist-test-slave-gp6n krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:35.954138 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:35.937939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.28.21.62:42999)
0806 02:07:35.938228 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:35.938239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:35.938342 (+ 103us) server_negotiation.cc:407] Connection header received
0806 02:07:35.938598 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:35.938603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:35.938659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:35.938765 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:35.939619 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:35.940333 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:35.941313 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:35.941574 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:35.941691 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:35.950999 (+ 9308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:35.951003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:35.951005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:35.951033 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:35.952482 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:35.952898 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:35.952901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:35.952903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:35.952989 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:35.953234 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:35.953236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:35.953237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:35.953439 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:35.953557 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:35.953826 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 02:07:35.953973 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260806 02:07:35.956825 30595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:35.957319 30595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:35.962059 30888 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 (DEFAULT_TABLE table=test-table [id=33b29b71eef64994a05be39ca2ea8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:35.962095 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 (DEFAULT_TABLE table=test-table [id=33b29b71eef64994a05be39ca2ea8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:35.962162 31032 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 (DEFAULT_TABLE table=test-table [id=33b29b71eef64994a05be39ca2ea8d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:35.962446 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9bab2449664b90b0d4a2d86d265c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:35.962453 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9bab2449664b90b0d4a2d86d265c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:35.962473 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9bab2449664b90b0d4a2d86d265c04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:35.964684 31225 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Bootstrap starting.
I20260806 02:07:35.964969 31228 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Bootstrap starting.
I20260806 02:07:35.966003 31228 tablet_bootstrap.cc:654] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:35.966003 31225 tablet_bootstrap.cc:654] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:35.966732 31227 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Bootstrap starting.
I20260806 02:07:35.967372 31228 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: No bootstrap required, opened a new log
I20260806 02:07:35.967469 31228 ts_tablet_manager.cc:1397] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:35.967767 31227 tablet_bootstrap.cc:654] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:35.967962 31228 raft_consensus.cc:348] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.968091 31228 raft_consensus.cc:374] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:35.968132 31228 raft_consensus.cc:729] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Initialized, Role: FOLLOWER
I20260806 02:07:35.968279 31228 consensus_queue.cc:260] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.968557 31228 ts_tablet_manager.cc:1428] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:35.969087 31227 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: No bootstrap required, opened a new log
I20260806 02:07:35.969087 31225 tablet_bootstrap.cc:492] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: No bootstrap required, opened a new log
I20260806 02:07:35.969175 31225 ts_tablet_manager.cc:1397] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:07:35.969229 31227 ts_tablet_manager.cc:1397] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:35.969635 31225 raft_consensus.cc:348] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.969650 31227 raft_consensus.cc:348] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.969758 31227 raft_consensus.cc:374] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:35.969758 31225 raft_consensus.cc:374] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:35.969794 31227 raft_consensus.cc:729] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Initialized, Role: FOLLOWER
I20260806 02:07:35.969803 31225 raft_consensus.cc:729] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Initialized, Role: FOLLOWER
I20260806 02:07:35.969930 31227 consensus_queue.cc:260] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.969931 31225 consensus_queue.cc:260] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:35.970290 31227 ts_tablet_manager.cc:1428] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:35.970368 31225 ts_tablet_manager.cc:1428] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:36.241237 31205 raft_consensus.cc:482] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:36.241385 31205 raft_consensus.cc:504] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.241904 31205 leader_election.cc:288] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:36.242548 30909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9bab2449664b90b0d4a2d86d265c04" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac" is_pre_election: true
I20260806 02:07:36.242543 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9bab2449664b90b0d4a2d86d265c04" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5" is_pre_election: true
I20260806 02:07:36.242734 30909 raft_consensus.cc:2454] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 0.
I20260806 02:07:36.242733 31054 raft_consensus.cc:2454] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 0.
I20260806 02:07:36.243108 30691 leader_election.cc:302] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccd39b6bfe4437d924d8f6cd939d835, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:36.243361 31205 raft_consensus.cc:2781] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:36.243424 31205 raft_consensus.cc:482] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:36.243455 31205 raft_consensus.cc:3037] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.244535 31205 raft_consensus.cc:504] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.244879 31205 leader_election.cc:288] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 election: Requested vote from peers 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:36.245188 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9bab2449664b90b0d4a2d86d265c04" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5"
I20260806 02:07:36.245182 30909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9bab2449664b90b0d4a2d86d265c04" candidate_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440ef057c5874fb590cd9c7c94fb0dac"
I20260806 02:07:36.245314 31054 raft_consensus.cc:3037] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.245335 30909 raft_consensus.cc:3037] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.246582 30909 raft_consensus.cc:2454] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 1.
I20260806 02:07:36.246582 31054 raft_consensus.cc:2454] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccd39b6bfe4437d924d8f6cd939d835 in term 1.
I20260806 02:07:36.246933 30692 leader_election.cc:302] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccd39b6bfe4437d924d8f6cd939d835, 440ef057c5874fb590cd9c7c94fb0dac; no voters:
I20260806 02:07:36.247149 31205 raft_consensus.cc:2781] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:36.247270 31205 raft_consensus.cc:686] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Running, Role: LEADER
I20260806 02:07:36.247433 31205 consensus_queue.cc:237] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.249276 30597 catalog_manager.cc:5495] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:36.269500 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.265794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.28.21.1:44023)
0806 02:07:36.266103 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.266129 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.266146 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:07:36.266193 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.266196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.266247 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.266351 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:36.266662 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.267250 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.268263 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.268564 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.268703 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.268990 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.269072 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.269161 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.269259 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.269330 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260806 02:07:36.275046 30909 raft_consensus.cc:1261] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Refusing update from remote peer 2ccd39b6bfe4437d924d8f6cd939d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:36.275269 31054 raft_consensus.cc:1261] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 2ccd39b6bfe4437d924d8f6cd939d835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:36.275512 31205 consensus_queue.cc:1035] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:36.275674 31206 consensus_queue.cc:1035] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:36.308000 30597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:36.308184 30597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:36.310149 30597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Sending DeleteTablet for 3 replicas of tablet 8b9bab2449664b90b0d4a2d86d265c04
I20260806 02:07:36.310752 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:36 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:36982
I20260806 02:07:36.310814 30888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:36 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:48624
I20260806 02:07:36.310925 31214 tablet_replica.cc:330] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: stopping tablet replica
I20260806 02:07:36.310990 31215 tablet_replica.cc:330] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: stopping tablet replica
I20260806 02:07:36.311053 31214 raft_consensus.cc:2229] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:36.311108 31215 raft_consensus.cc:2229] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:36.311223 31215 raft_consensus.cc:2258] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:36.311249 31214 raft_consensus.cc:2258] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:36.311393 30597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:07:36.311787 31032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9bab2449664b90b0d4a2d86d265c04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:36 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51346
W20260806 02:07:36.311817 30597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:36.311900 31215 ts_tablet_manager.cc:1905] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:36.311968 31216 tablet_replica.cc:330] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: stopping tablet replica
I20260806 02:07:36.312074 31216 raft_consensus.cc:2229] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:36.312182 31216 raft_consensus.cc:2258] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:36.312409 31214 ts_tablet_manager.cc:1905] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:36.314368 31215 ts_tablet_manager.cc:1918] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:36.314484 31215 log.cc:1196] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/8b9bab2449664b90b0d4a2d86d265c04
I20260806 02:07:36.314778 31215 ts_tablet_manager.cc:1939] T 8b9bab2449664b90b0d4a2d86d265c04 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting consensus metadata
I20260806 02:07:36.315011 31214 ts_tablet_manager.cc:1918] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:36.315084 31214 log.cc:1196] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/8b9bab2449664b90b0d4a2d86d265c04
I20260806 02:07:36.315368 31214 ts_tablet_manager.cc:1939] T 8b9bab2449664b90b0d4a2d86d265c04 P 2ccd39b6bfe4437d924d8f6cd939d835: Deleting consensus metadata
I20260806 02:07:36.315531 30583 catalog_manager.cc:4841] TS 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2:36917): tablet 8b9bab2449664b90b0d4a2d86d265c04 (table test-table [id=33b29b71eef64994a05be39ca2ea8d6d]) successfully deleted
I20260806 02:07:36.316102 30580 catalog_manager.cc:4841] TS 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023): tablet 8b9bab2449664b90b0d4a2d86d265c04 (table test-table [id=33b29b71eef64994a05be39ca2ea8d6d]) successfully deleted
I20260806 02:07:36.316496 30888 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca3b69920494334b25e829f03e30c47 (DEFAULT_TABLE table=test-table [id=7de5dd9206e14270be46387c03ce03e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:36.316540 30741 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca3b69920494334b25e829f03e30c47 (DEFAULT_TABLE table=test-table [id=7de5dd9206e14270be46387c03ce03e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:36.316736 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca3b69920494334b25e829f03e30c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:36.316776 30741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca3b69920494334b25e829f03e30c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:36.316718 31032 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca3b69920494334b25e829f03e30c47 (DEFAULT_TABLE table=test-table [id=7de5dd9206e14270be46387c03ce03e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:36.316990 31032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca3b69920494334b25e829f03e30c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:36.318862 31225 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: Bootstrap starting.
I20260806 02:07:36.318928 31228 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: Bootstrap starting.
I20260806 02:07:36.316726 31216 ts_tablet_manager.cc:1905] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:36.319409 31227 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Bootstrap starting.
I20260806 02:07:36.319777 31228 tablet_bootstrap.cc:654] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:36.319802 31225 tablet_bootstrap.cc:654] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:36.320250 31227 tablet_bootstrap.cc:654] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:36.321069 31225 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: No bootstrap required, opened a new log
I20260806 02:07:36.321166 31225 ts_tablet_manager.cc:1397] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 02:07:36.321449 31227 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: No bootstrap required, opened a new log
I20260806 02:07:36.321537 31227 ts_tablet_manager.cc:1397] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:36.321626 31225 raft_consensus.cc:348] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.321740 31225 raft_consensus.cc:374] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:36.321785 31225 raft_consensus.cc:729] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Initialized, Role: FOLLOWER
I20260806 02:07:36.321789 31216 ts_tablet_manager.cc:1918] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:07:36.321869 31216 log.cc:1196] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/8b9bab2449664b90b0d4a2d86d265c04
I20260806 02:07:36.321910 31225 consensus_queue.cc:260] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.321926 31227 raft_consensus.cc:348] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.322031 31227 raft_consensus.cc:374] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:36.322072 31227 raft_consensus.cc:729] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b57ba571a14150a52b12e209bcd5a5, State: Initialized, Role: FOLLOWER
I20260806 02:07:36.322149 31216 ts_tablet_manager.cc:1939] T 8b9bab2449664b90b0d4a2d86d265c04 P c5b57ba571a14150a52b12e209bcd5a5: Deleting consensus metadata
I20260806 02:07:36.322196 31225 ts_tablet_manager.cc:1428] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:07:36.322196 31227 consensus_queue.cc:260] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.322472 31227 ts_tablet_manager.cc:1428] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:36.322954 30582 catalog_manager.cc:4841] TS c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781): tablet 8b9bab2449664b90b0d4a2d86d265c04 (table test-table [id=33b29b71eef64994a05be39ca2ea8d6d]) successfully deleted
I20260806 02:07:36.323958 31228 tablet_bootstrap.cc:492] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: No bootstrap required, opened a new log
I20260806 02:07:36.324061 31228 ts_tablet_manager.cc:1397] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 02:07:36.324481 31228 raft_consensus.cc:348] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.324615 31228 raft_consensus.cc:374] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:36.324656 31228 raft_consensus.cc:729] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccd39b6bfe4437d924d8f6cd939d835, State: Initialized, Role: FOLLOWER
I20260806 02:07:36.324780 31228 consensus_queue.cc:260] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.325035 31228 ts_tablet_manager.cc:1428] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:36.609561 31232 raft_consensus.cc:482] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:36.609752 31232 raft_consensus.cc:504] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.611078 31232 leader_election.cc:288] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:36.615459 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.611296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.28.21.2:48049)
0806 02:07:36.611630 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:36.611646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:36.611782 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:36.612113 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:36.612118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:36.612139 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:36.612305 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.612312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.613194 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.613197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.614067 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.614076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.614971 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.614977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.615084 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.615113 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:07:36.615216 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:07:36.615288 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260806 02:07:36.615516 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.611441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:48049 (local address 127.28.21.1:44023)
0806 02:07:36.611578 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.611585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.611776 (+ 191us) server_negotiation.cc:407] Connection header received
0806 02:07:36.611919 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.611924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.611988 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.612072 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:36.612440 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.613029 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.614209 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.614830 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.614958 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.615039 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.615279 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.615368 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:07:36.615651 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.611340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.28.21.2:33287)
0806 02:07:36.611706 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:36.611744 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:36.611881 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:36.612188 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:36.612193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:36.612212 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:36.612359 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.612391 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.613180 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.613184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.614171 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.614185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.615123 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.615126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.615318 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.615343 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:07:36.615425 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:07:36.615491 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":48,"thread_start_us":124,"threads_started":1}
I20260806 02:07:36.616015 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.611447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:33287 (local address 127.28.21.3:36781)
0806 02:07:36.611672 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.611680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.611866 (+ 186us) server_negotiation.cc:407] Connection header received
0806 02:07:36.611993 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.611996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.612039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.612147 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:36.612514 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.613032 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.614308 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.614916 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.615026 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.615092 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.615773 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.615868 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260806 02:07:36.616302 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca3b69920494334b25e829f03e30c47" candidate_uuid: "440ef057c5874fb590cd9c7c94fb0dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" is_pre_election: true
I20260806 02:07:36.616468 30761 raft_consensus.cc:2454] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440ef057c5874fb590cd9c7c94fb0dac in term 0.
I20260806 02:07:36.616493 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca3b69920494334b25e829f03e30c47" candidate_uuid: "440ef057c5874fb590cd9c7c94fb0dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5" is_pre_election: true
I20260806 02:07:36.616683 31054 raft_consensus.cc:2454] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440ef057c5874fb590cd9c7c94fb0dac in term 0.
I20260806 02:07:36.616906 30835 leader_election.cc:302] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccd39b6bfe4437d924d8f6cd939d835, 440ef057c5874fb590cd9c7c94fb0dac; no voters:
I20260806 02:07:36.617164 31232 raft_consensus.cc:2781] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:36.617224 31232 raft_consensus.cc:482] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:36.617254 31232 raft_consensus.cc:3037] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.618614 31232 raft_consensus.cc:504] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.618957 31232 leader_election.cc:288] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [CANDIDATE]: Term 1 election: Requested vote from peers 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023), c5b57ba571a14150a52b12e209bcd5a5 (127.28.21.3:36781)
I20260806 02:07:36.619309 31054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca3b69920494334b25e829f03e30c47" candidate_uuid: "440ef057c5874fb590cd9c7c94fb0dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b57ba571a14150a52b12e209bcd5a5"
I20260806 02:07:36.619279 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca3b69920494334b25e829f03e30c47" candidate_uuid: "440ef057c5874fb590cd9c7c94fb0dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccd39b6bfe4437d924d8f6cd939d835"
I20260806 02:07:36.619432 31054 raft_consensus.cc:3037] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.619465 30761 raft_consensus.cc:3037] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:36.620456 31054 raft_consensus.cc:2454] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440ef057c5874fb590cd9c7c94fb0dac in term 1.
I20260806 02:07:36.620568 30761 raft_consensus.cc:2454] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440ef057c5874fb590cd9c7c94fb0dac in term 1.
I20260806 02:07:36.620769 30837 leader_election.cc:302] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440ef057c5874fb590cd9c7c94fb0dac, c5b57ba571a14150a52b12e209bcd5a5; no voters:
I20260806 02:07:36.620947 31232 raft_consensus.cc:2781] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:36.621168 31232 raft_consensus.cc:686] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 LEADER]: Becoming Leader. State: Replica: 440ef057c5874fb590cd9c7c94fb0dac, State: Running, Role: LEADER
I20260806 02:07:36.621435 31232 consensus_queue.cc:237] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } }
I20260806 02:07:36.623997 30596 catalog_manager.cc:5495] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac reported cstate change: term changed from 0 to 1, leader changed from <none> to 440ef057c5874fb590cd9c7c94fb0dac (127.28.21.2). New cstate: current_term: 1 leader_uuid: "440ef057c5874fb590cd9c7c94fb0dac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440ef057c5874fb590cd9c7c94fb0dac" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:36.700922 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.696854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.28.21.2:36917)
0806 02:07:36.697184 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.697191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.697202 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:07:36.697274 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.697276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.697328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.697443 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:36.697655 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.698212 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.699118 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.699356 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.699454 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.699857 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.700322 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.700419 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.700592 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.700648 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":175,"threads_started":1}
I20260806 02:07:36.711741 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.708295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:37457 (local address 127.28.21.3:36781)
0806 02:07:36.708417 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.708422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.708440 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:07:36.708568 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.708571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.708615 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.708680 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:36.708940 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.709658 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.710605 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.710829 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.710908 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.711233 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.711375 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.711446 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.711589 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.711636 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:07:36.711760 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.708068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:36781 (local address 127.28.21.2:37457)
0806 02:07:36.708379 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:36.708400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:36.708480 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:36.708713 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:36.708715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:36.708721 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:36.708840 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.708845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.709781 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.709783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.710498 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.710507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.710956 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.710959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.711035 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.711081 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.711487 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.711494 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:07:36.711565 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:07:36.711634 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":112,"threads_started":1}
I20260806 02:07:36.717967 30761 raft_consensus.cc:1261] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835 [term 1 FOLLOWER]: Refusing update from remote peer 440ef057c5874fb590cd9c7c94fb0dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:36.718132 31054 raft_consensus.cc:1261] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 440ef057c5874fb590cd9c7c94fb0dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:36.718580 31246 consensus_queue.cc:1035] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccd39b6bfe4437d924d8f6cd939d835" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:36.718777 31232 consensus_queue.cc:1035] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b57ba571a14150a52b12e209bcd5a5" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:36.736796 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.733052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36917 (local address 127.28.21.3:38601)
0806 02:07:36.733352 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:36.733365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:36.733491 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:36.733775 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:36.733778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:36.733787 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:07:36.733891 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.733897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.734681 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.734690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.735304 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:36.735313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.735889 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.735892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:36.735989 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.736021 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.736455 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.736464 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:07:36.736537 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:07:36.736657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
I20260806 02:07:36.736917 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:36.733128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:38601 (local address 127.28.21.2:36917)
0806 02:07:36.733281 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:36.733285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:36.733447 (+ 162us) server_negotiation.cc:407] Connection header received
0806 02:07:36.733628 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:36.733631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:36.733679 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:36.733764 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:36.734002 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.734578 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.735432 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:36.735667 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:36.735783 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:36.736216 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:36.736311 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:36.736422 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:36.736679 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:07:36.736742 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:07:36.737564 31195 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:07:38.297694 30592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:38.297894 30592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:38.300360 30592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61d528c642f40338bb8c2b4353ec204: Sending DeleteTablet for 3 replicas of tablet 0ca3b69920494334b25e829f03e30c47
I20260806 02:07:38.300964 30888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca3b69920494334b25e829f03e30c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:38 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:48624
I20260806 02:07:38.301007 30741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca3b69920494334b25e829f03e30c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:38 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:36982
I20260806 02:07:38.301044 31032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca3b69920494334b25e829f03e30c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:38 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:51346
I20260806 02:07:38.301373 31283 tablet_replica.cc:330] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: stopping tablet replica
I20260806 02:07:38.301491 31284 tablet_replica.cc:330] T 0ca3b69920494334b25e829f03e30c47 P 2ccd39b6bfe4437d924d8f6cd939d835: stopping tablet replica
I20260806 02:07:38.301537 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30671
I20260806 02:07:38.301533 31283 raft_consensus.cc:2229] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:38.303671 31283 raft_consensus.cc:2258] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:38.304574 31283 ts_tablet_manager.cc:1905] T 0ca3b69920494334b25e829f03e30c47 P 440ef057c5874fb590cd9c7c94fb0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:38.304592 31285 tablet_replica.cc:330] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: stopping tablet replica
I20260806 02:07:38.304723 31285 raft_consensus.cc:2229] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:38.304905 31285 raft_consensus.cc:2258] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:38.305553 31285 ts_tablet_manager.cc:1905] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:07:38.310796 30580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:44023 (error 108)
I20260806 02:07:38.310885 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30819
I20260806 02:07:38.313643 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:38.312253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.0.0.1:37006)
0806 02:07:38.313424 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:38.313524 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":204,"threads_started":1}
W20260806 02:07:38.313817 30580 catalog_manager.cc:4584] TS 2ccd39b6bfe4437d924d8f6cd939d835 (127.28.21.1:44023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ca3b69920494334b25e829f03e30c47: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44023: connect: Connection refused (error 111)
I20260806 02:07:38.367328 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:38.366977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.0.0.1:37010)
0806 02:07:38.367149 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:38.367228 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:07:38.368112 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:38.367885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44023 (local address 127.0.0.1:37016)
0806 02:07:38.368000 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:38.368046 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 02:07:38.392932 31285 ts_tablet_manager.cc:1918] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:07:38.393100 31285 log.cc:1196] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/0ca3b69920494334b25e829f03e30c47
I20260806 02:07:38.393425 31285 ts_tablet_manager.cc:1939] T 0ca3b69920494334b25e829f03e30c47 P c5b57ba571a14150a52b12e209bcd5a5: Deleting consensus metadata
I20260806 02:07:38.398444 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30965
I20260806 02:07:38.401199 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:38.399970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36917 (local address 127.0.0.1:48652)
0806 02:07:38.401016 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:38.401083 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.2:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":929}
I20260806 02:07:38.409745 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 30564
2026-08-06T02:07:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6773 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:38 dist-test-slave-gp6n krb5kdc[31290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:38 dist-test-slave-gp6n krb5kdc[31290](info): set up 2 sockets
Aug 06 02:07:38 dist-test-slave-gp6n krb5kdc[31290](info): commencing operation
krb5kdc: starting...
W20260806 02:07:39.485388 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:39 dist-test-slave-gp6n krb5kdc[31290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:39.613871 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43883
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:43883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:39.723670 31303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:39.723946 31303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:39.724030 31303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:39.728025 31303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:39.728080 31303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:39.728101 31303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:39.728117 31303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:39.728174 31303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:39.733551 31303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:43883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:39.734635 31303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:39.735709 31303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:39.744571 31309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:39.744520 31308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:39.744550 31303 server_base.cc:1034] running on GCE node
W20260806 02:07:39.744521 31311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:39.745397 31303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:39.746531 31303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:39.747711 31303 hybrid_clock.cc:648] HybridClock initialized: now 1785982059747701 us; error 32 us; skew 500 ppm
Aug 06 02:07:39 dist-test-slave-gp6n krb5kdc[31290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982059, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:39.750487 31303 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:39.751323 31303 webserver.cc:460] Webserver started at http://127.28.21.62:44771/ using document root <none> and password file <none>
I20260806 02:07:39.751884 31303 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:39.751976 31303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:39.752209 31303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:39.754261 31303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "41e8ab84bb274793a0fe6b03f10f4194"
format_stamp: "Formatted at 2026-08-06 02:07:39 on dist-test-slave-gp6n"
server_key: "090f1251a7f3bf02a51d8654b8b21f92"
server_key_iv: "be21ee265fd01649a22d0039367d7a45"
server_key_version: "encryptionkey@0"
I20260806 02:07:39.754907 31303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "41e8ab84bb274793a0fe6b03f10f4194"
format_stamp: "Formatted at 2026-08-06 02:07:39 on dist-test-slave-gp6n"
server_key: "090f1251a7f3bf02a51d8654b8b21f92"
server_key_iv: "be21ee265fd01649a22d0039367d7a45"
server_key_version: "encryptionkey@0"
I20260806 02:07:39.758599 31303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:39.760896 31318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:39.762182 31303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:39.762336 31303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "41e8ab84bb274793a0fe6b03f10f4194"
format_stamp: "Formatted at 2026-08-06 02:07:39 on dist-test-slave-gp6n"
server_key: "090f1251a7f3bf02a51d8654b8b21f92"
server_key_iv: "be21ee265fd01649a22d0039367d7a45"
server_key_version: "encryptionkey@0"
I20260806 02:07:39.762529 31303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:39.781183 31303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:39.784312 31303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:39.784535 31303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:39.791678 31303 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:43883
I20260806 02:07:39.791704 31374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:43883 every 8 connection(s)
I20260806 02:07:39.792654 31303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:39.795367 31375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:39.800194 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31303
I20260806 02:07:39.800311 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:39.800597 28756 external_mini_cluster.cc:1428] Setting key 2325387b8dd995288f37ac7e929835b8
I20260806 02:07:39.801131 31375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: Bootstrap starting.
I20260806 02:07:39.803565 31375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:39.804571 31375 log.cc:824] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:39.806710 31375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: No bootstrap required, opened a new log
Aug 06 02:07:39 dist-test-slave-gp6n krb5kdc[31290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:39.811254 31375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } }
I20260806 02:07:39.811563 31375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:39.811630 31375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e8ab84bb274793a0fe6b03f10f4194, State: Initialized, Role: FOLLOWER
I20260806 02:07:39.812124 31375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } }
I20260806 02:07:39.812288 31375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:39.812369 31375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:39.812476 31375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:39.813568 31375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } }
I20260806 02:07:39.813915 31375 leader_election.cc:302] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41e8ab84bb274793a0fe6b03f10f4194; no voters:
I20260806 02:07:39.814260 31375 leader_election.cc:288] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:39.814316 31380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:39.814750 31380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [term 1 LEADER]: Becoming Leader. State: Replica: 41e8ab84bb274793a0fe6b03f10f4194, State: Running, Role: LEADER
I20260806 02:07:39.815100 31380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } }
I20260806 02:07:39.815636 31375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:39.815771 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:39.802022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.28.21.62:43883)
0806 02:07:39.802525 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:39.802540 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:39.802577 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:07:39.803193 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:39.803208 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:39.803536 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:39.803871 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:39.804821 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:39.805983 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:39.807147 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:39.807621 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:39.807739 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:39.810582 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:39.810589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:39.810604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:39.810635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:39.812616 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:39.813024 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:39.813030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:39.813036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:39.813178 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:39.813446 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:39.813452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:39.813457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:39.813971 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:39.814161 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:39.814551 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:07:39.814892 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260806 02:07:39.817698 31381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41e8ab84bb274793a0fe6b03f10f4194. Latest consensus state: current_term: 1 leader_uuid: "41e8ab84bb274793a0fe6b03f10f4194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } } }
I20260806 02:07:39.817826 31381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:39.817904 31382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41e8ab84bb274793a0fe6b03f10f4194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e8ab84bb274793a0fe6b03f10f4194" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43883 } } }
I20260806 02:07:39.817982 31382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194 [sys.catalog]: This master's current role is: LEADER
W20260806 02:07:39.820431 31394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:39.820493 31394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:39.820571 31395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:39.821288 31395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:39.826376 31395 catalog_manager.cc:1347] Generated new cluster ID: 2aeb5a58ba06472fa8c1cc2cbc1605f2
I20260806 02:07:39.826473 31395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:39.836491 31395 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:39.837477 31395 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:39.840875 31395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41e8ab84bb274793a0fe6b03f10f4194: Generated new TSK 0
I20260806 02:07:39.841535 31395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:39.873317 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:39.986653 31401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:39.986896 31401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:39.986937 31401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:39.990869 31401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:39.990921 31401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:39.990978 31401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:39.995987 31401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:39.997121 31401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:39.998312 31401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:40.007977 31408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:40.007972 31407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:40.007972 31410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:40.008308 31401 server_base.cc:1034] running on GCE node
I20260806 02:07:40.008729 31401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:40.009470 31401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:40.010664 31401 hybrid_clock.cc:648] HybridClock initialized: now 1785982060010635 us; error 39 us; skew 500 ppm
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:40.013480 31401 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:40.014376 31401 webserver.cc:460] Webserver started at http://127.28.21.1:34209/ using document root <none> and password file <none>
I20260806 02:07:40.015003 31401 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:40.015100 31401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:40.015340 31401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:40.017591 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "697b1c1e2dda4ee2a658561e05c31e7c"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "ec84e34d158dc6789db350a1598694ad"
server_key_iv: "8c1a5f01163df7d7d661502e40b7b88c"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.018172 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "697b1c1e2dda4ee2a658561e05c31e7c"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "ec84e34d158dc6789db350a1598694ad"
server_key_iv: "8c1a5f01163df7d7d661502e40b7b88c"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.021960 31401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:40.024551 31417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.025811 31401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:40.026053 31401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "697b1c1e2dda4ee2a658561e05c31e7c"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "ec84e34d158dc6789db350a1598694ad"
server_key_iv: "8c1a5f01163df7d7d661502e40b7b88c"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.026166 31401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:40.041563 31401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:40.044829 31401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:40.045053 31401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:40.045692 31401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:40.046592 31401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:40.046660 31401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.046731 31401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:40.046773 31401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.056242 31401 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:43029
I20260806 02:07:40.056281 31538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:43029 every 8 connection(s)
I20260806 02:07:40.057191 31401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:40.060065 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31401
I20260806 02:07:40.060200 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:40.060513 28756 external_mini_cluster.cc:1428] Setting key c6aec9673fa7ec52b7997a8b73acbe87
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:40.073225 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.059319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:57465 (local address 127.28.21.62:43883)
0806 02:07:40.059491 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:40.059496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:40.060736 (+ 1240us) server_negotiation.cc:407] Connection header received
0806 02:07:40.061667 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:40.061672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:40.061722 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:40.061811 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:40.063153 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.064006 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.065873 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.066291 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.066450 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.069306 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:40.069310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:40.069313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:40.069343 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:40.071181 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.071725 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.071729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.071732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.071832 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.072160 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.072162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.072163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.072332 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:40.072407 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:40.072981 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 02:07:40.073102 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:07:40.074069 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.059730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43883 (local address 127.28.21.1:57465)
0806 02:07:40.060554 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:40.060612 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:40.061469 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:40.062021 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:40.062028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:40.062487 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:40.063001 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.063016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.064183 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.064188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.065751 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.065761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.066541 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.066545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.066762 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.067395 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:40.067411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:40.069123 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:40.071294 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.071299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.071311 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.071622 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.071940 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.071942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.071944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.072076 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.072429 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:40.072434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:40.072707 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 02:07:40.072906 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:07:40.073155 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:40.075512 31539 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43883
I20260806 02:07:40.075752 31539 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:40.076231 31539 heartbeater.cc:511] Master 127.28.21.62:43883 requested a full tablet report, sending...
I20260806 02:07:40.077706 31335 ts_manager.cc:194] Registered new tserver with Master: 697b1c1e2dda4ee2a658561e05c31e7c (127.28.21.1:43029)
I20260806 02:07:40.078887 31335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:57465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:40.089747 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:40.206435 31544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:40.206666 31544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:40.206709 31544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:40.210709 31544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:40.210768 31544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:40.210824 31544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:40.216015 31544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:40.217340 31544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:40.218783 31544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:40.230383 31554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:40.230383 31550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:40.230381 31551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:40.230974 31544 server_base.cc:1034] running on GCE node
I20260806 02:07:40.231388 31544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:40.232024 31544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:40.233213 31544 hybrid_clock.cc:648] HybridClock initialized: now 1785982060233175 us; error 65 us; skew 500 ppm
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:40.236155 31544 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:40.237025 31544 webserver.cc:460] Webserver started at http://127.28.21.2:42659/ using document root <none> and password file <none>
I20260806 02:07:40.237586 31544 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:40.237679 31544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:40.237920 31544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:40.240031 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "ed0f7a4168c346c6a0af2120b5611ea2"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "e27637b1cbba40bea5136876129a1e06"
server_key_iv: "c18272ea6448715e233e173612d5a949"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.240594 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0f7a4168c346c6a0af2120b5611ea2"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "e27637b1cbba40bea5136876129a1e06"
server_key_iv: "c18272ea6448715e233e173612d5a949"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.244383 31544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:40.247023 31561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.248281 31544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:40.248430 31544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "ed0f7a4168c346c6a0af2120b5611ea2"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "e27637b1cbba40bea5136876129a1e06"
server_key_iv: "c18272ea6448715e233e173612d5a949"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.248540 31544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:40.263051 31544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:40.266554 31544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:40.266755 31544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:40.267411 31544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:40.268246 31544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:40.268318 31544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.268384 31544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:40.268435 31544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.277311 31544 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:40987
I20260806 02:07:40.277379 31685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:40987 every 8 connection(s)
I20260806 02:07:40.278239 31544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:40.286854 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31544
I20260806 02:07:40.286985 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:40.287340 28756 external_mini_cluster.cc:1428] Setting key c85c1d9be1906a948f39425c38b0342c
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:40.293043 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.280121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:47577 (local address 127.28.21.62:43883)
0806 02:07:40.280260 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:40.280263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:40.281311 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 02:07:40.282220 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:40.282223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:40.282283 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:40.282394 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:40.283574 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.284106 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.285284 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.285519 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.285633 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.288798 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:40.288803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:40.288807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:40.288839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:40.290667 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.291225 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.291231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.291234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.291355 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.291708 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.291712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.291713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.291986 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:40.292107 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:40.292625 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:07:40.292780 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:07:40.293689 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.280502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43883 (local address 127.28.21.2:47577)
0806 02:07:40.281163 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:40.281203 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:40.282014 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:40.282594 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:40.282601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:40.282957 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:40.283429 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.283442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.284235 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.284238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.285148 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.285157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.285672 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.285676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.285844 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.286551 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:40.286589 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:40.288529 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:40.290814 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.290824 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.290843 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.291118 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.291472 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.291475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.291476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.291612 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.292110 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:40.292115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:40.292350 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 02:07:40.292530 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:07:40.292789 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":105,"threads_started":1}
I20260806 02:07:40.294893 31686 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43883
I20260806 02:07:40.295158 31686 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:40.295647 31686 heartbeater.cc:511] Master 127.28.21.62:43883 requested a full tablet report, sending...
I20260806 02:07:40.296726 31335 ts_manager.cc:194] Registered new tserver with Master: ed0f7a4168c346c6a0af2120b5611ea2 (127.28.21.2:40987)
I20260806 02:07:40.297300 31335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:47577
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:40.317833 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:40.431461 31692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:40.431716 31692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:40.431780 31692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:40.435762 31692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:40.435817 31692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:40.435904 31692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:40.441061 31692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:40.442065 31692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:40.443104 31692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:40.452770 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:40.452746 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:40.452806 31692 server_base.cc:1034] running on GCE node
W20260806 02:07:40.452746 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:40.453369 31692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:40.454059 31692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:40.455214 31692 hybrid_clock.cc:648] HybridClock initialized: now 1785982060455188 us; error 37 us; skew 500 ppm
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:40.458117 31692 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:40.459048 31692 webserver.cc:460] Webserver started at http://127.28.21.3:44937/ using document root <none> and password file <none>
I20260806 02:07:40.459590 31692 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:40.459656 31692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:40.459836 31692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:40.461970 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "d4f484a062a04365a46b856e5ef0fc8b"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "0bbb5730f0d473cd33c172a48f2aef00"
server_key_iv: "14150e9dc73a6c804d4c7f5da66c7de8"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.462615 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "d4f484a062a04365a46b856e5ef0fc8b"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "0bbb5730f0d473cd33c172a48f2aef00"
server_key_iv: "14150e9dc73a6c804d4c7f5da66c7de8"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.466892 31692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:07:40.469578 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.470916 31692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:40.471027 31692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "d4f484a062a04365a46b856e5ef0fc8b"
format_stamp: "Formatted at 2026-08-06 02:07:40 on dist-test-slave-gp6n"
server_key: "0bbb5730f0d473cd33c172a48f2aef00"
server_key_iv: "14150e9dc73a6c804d4c7f5da66c7de8"
server_key_version: "encryptionkey@0"
I20260806 02:07:40.471117 31692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:40.487223 31692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:40.490336 31692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:40.490576 31692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:40.491163 31692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:40.491951 31692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:40.492009 31692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.492050 31692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:40.492065 31692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:40.501107 31692 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:34989
I20260806 02:07:40.501127 31829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:34989 every 8 connection(s)
I20260806 02:07:40.502009 31692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:40.504645 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31692
I20260806 02:07:40.504776 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:40.505091 28756 external_mini_cluster.cc:1428] Setting key 21917d1adafe59e719eb588ea500c52a
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:40.517432 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.503925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:44839 (local address 127.28.21.62:43883)
0806 02:07:40.504077 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:40.504081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:40.505442 (+ 1361us) server_negotiation.cc:407] Connection header received
0806 02:07:40.506443 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:40.506447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:40.506506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:40.506630 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:40.508072 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.508665 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.510007 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.510261 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.510428 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.513258 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:40.513265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:40.513273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:40.513298 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:40.514955 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.515594 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.515598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.515600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.515692 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:40.516083 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:40.516087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:40.516089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:40.516289 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:40.516445 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:40.517070 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:07:40.517227 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:07:40.518558 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:40.504332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43883 (local address 127.28.21.3:44839)
0806 02:07:40.505260 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:40.505306 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:40.506178 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:40.506823 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:40.506830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:40.507334 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:40.507899 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.507916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.508812 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.508816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.509855 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:40.509870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:40.510485 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:40.510489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:40.510688 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:40.511380 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:40.511398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:40.513066 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:40.515102 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.515108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.515126 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.515478 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.515794 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:40.515799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:40.515801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:40.515963 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:40.516458 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:40.516467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:40.516789 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 02:07:40.517010 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 02:07:40.517367 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":163,"threads_started":1}
I20260806 02:07:40.519874 31831 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43883
I20260806 02:07:40.520141 31831 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:40.520668 31831 heartbeater.cc:511] Master 127.28.21.62:43883 requested a full tablet report, sending...
I20260806 02:07:40.521736 31335 ts_manager.cc:194] Registered new tserver with Master: d4f484a062a04365a46b856e5ef0fc8b (127.28.21.3:34989)
I20260806 02:07:40.522310 31335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:44839
I20260806 02:07:40.531311 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:07:40.546211 31378 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:40.544817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.28.21.62:43883)
0806 02:07:40.544969 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:40.544973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:40.545120 (+ 147us) server_negotiation.cc:407] Connection header received
0806 02:07:40.545243 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:40.545246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:40.545289 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:40.545370 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:40.546098 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44110: BlockingRecv error: recv got EOF from 127.0.0.1:44110 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:40.547598 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31401
I20260806 02:07:40.555702 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31544
I20260806 02:07:40.562086 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31692
I20260806 02:07:40.569114 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31303
2026-08-06T02:07:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2157 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31843](info): set up 2 sockets
Aug 06 02:07:40 dist-test-slave-gp6n krb5kdc[31843](info): commencing operation
krb5kdc: starting...
W20260806 02:07:41.637144 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:41 dist-test-slave-gp6n krb5kdc[31843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:41.772032 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:33611
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:41.911162 31856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:41.911412 31856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:41.911451 31856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:41.915356 31856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:41.915419 31856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:41.915439 31856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:41.915457 31856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:41.915472 31856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:41.920595 31856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:41.921664 31856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:41.922703 31856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:41.931268 31861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:41.931267 31862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:41.931267 31864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:41.931650 31856 server_base.cc:1034] running on GCE node
I20260806 02:07:41.932175 31856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:41.933254 31856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:41.934481 31856 hybrid_clock.cc:648] HybridClock initialized: now 1785982061934462 us; error 35 us; skew 500 ppm
Aug 06 02:07:41 dist-test-slave-gp6n krb5kdc[31843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:41.942598 31856 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:41.943490 31856 webserver.cc:460] Webserver started at http://127.28.21.62:40893/ using document root <none> and password file <none>
I20260806 02:07:41.944053 31856 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:41.944167 31856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:41.944407 31856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:41.946499 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "79d9a8309ae049ffbbbc837dd3c229c4"
format_stamp: "Formatted at 2026-08-06 02:07:41 on dist-test-slave-gp6n"
server_key: "dca36726c6583ea29d9ecf5f4c4b61af"
server_key_iv: "5cfebeef0675d476d28c5d6ca94e1136"
server_key_version: "encryptionkey@0"
I20260806 02:07:41.947069 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "79d9a8309ae049ffbbbc837dd3c229c4"
format_stamp: "Formatted at 2026-08-06 02:07:41 on dist-test-slave-gp6n"
server_key: "dca36726c6583ea29d9ecf5f4c4b61af"
server_key_iv: "5cfebeef0675d476d28c5d6ca94e1136"
server_key_version: "encryptionkey@0"
I20260806 02:07:41.950733 31856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:07:41.953111 31871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:41.954246 31856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:41.954438 31856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "79d9a8309ae049ffbbbc837dd3c229c4"
format_stamp: "Formatted at 2026-08-06 02:07:41 on dist-test-slave-gp6n"
server_key: "dca36726c6583ea29d9ecf5f4c4b61af"
server_key_iv: "5cfebeef0675d476d28c5d6ca94e1136"
server_key_version: "encryptionkey@0"
I20260806 02:07:41.954555 31856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:41.968789 31856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:41.971794 31856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:41.972012 31856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:41.979022 31856 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:33611
I20260806 02:07:41.979346 31929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:33611 every 8 connection(s)
I20260806 02:07:41.979984 31856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:41.982705 31930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:41.987977 31930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: Bootstrap starting.
I20260806 02:07:41.990195 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31856
I20260806 02:07:41.990319 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:41.990495 31930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:41.990669 28756 external_mini_cluster.cc:1428] Setting key f6894d0cec721488b7b4e57566614b85
I20260806 02:07:41.991454 31930 log.cc:824] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:41.993482 31930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: No bootstrap required, opened a new log
I20260806 02:07:41.996984 31930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } }
I20260806 02:07:41.997225 31930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:41.997316 31930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d9a8309ae049ffbbbc837dd3c229c4, State: Initialized, Role: FOLLOWER
I20260806 02:07:41.997835 31930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } }
I20260806 02:07:41.997989 31930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:41.998057 31930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:41.998155 31930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:07:41 dist-test-slave-gp6n krb5kdc[31843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:41.999390 31930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } }
I20260806 02:07:41.999799 31930 leader_election.cc:302] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d9a8309ae049ffbbbc837dd3c229c4; no voters:
I20260806 02:07:42.000136 31930 leader_election.cc:288] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:42.000285 31935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:42.000550 31935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [term 1 LEADER]: Becoming Leader. State: Replica: 79d9a8309ae049ffbbbc837dd3c229c4, State: Running, Role: LEADER
I20260806 02:07:42.000905 31935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } }
I20260806 02:07:42.001560 31930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:42.002635 31936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d9a8309ae049ffbbbc837dd3c229c4. Latest consensus state: current_term: 1 leader_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } } }
I20260806 02:07:42.002763 31936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:42.003160 31937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d9a8309ae049ffbbbc837dd3c229c4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 33611 } } }
I20260806 02:07:42.004432 31937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:42.006086 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:41.991722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.28.21.62:33611)
0806 02:07:41.992172 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:41.992188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:41.992223 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:07:41.992830 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:41.992846 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:41.993164 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:41.993500 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:41.994322 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:41.995359 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:41.996236 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:41.996708 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:41.996883 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:41.999816 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:41.999822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:41.999835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:41.999860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:42.002078 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:42.002643 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:42.002649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:42.002655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:42.002793 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:42.003138 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:42.003142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:42.003144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:42.003763 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:42.004003 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:42.004837 (+ 834us) server_negotiation.cc:299] Negotiation successful
0806 02:07:42.005115 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":155,"threads_started":1}
W20260806 02:07:42.006596 31951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:42.006681 31951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:42.006762 31945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:42.007617 31945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:42.009212 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:07:42.013634 31945 catalog_manager.cc:1347] Generated new cluster ID: 5cde64bbc1ee4fc6ae2095d0aac16d8e
I20260806 02:07:42.013698 31945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:42.034061 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:42.028208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.28.21.62:33611)
0806 02:07:42.028318 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:42.028323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:42.028599 (+ 276us) server_negotiation.cc:407] Connection header received
0806 02:07:42.028715 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:42.028719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:42.028774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:42.028865 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:42.029447 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:42.030280 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:42.031473 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:42.031845 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:42.031953 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:42.032954 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:42.032962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:42.032964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:42.033488 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:42.033531 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:42.033552 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:42.033644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:42.033831 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:07:42.033905 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:07:42.034444 31945 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:42.035668 31945 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:42.056708 31945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d9a8309ae049ffbbbc837dd3c229c4: Generated new TSK 0
I20260806 02:07:42.057529 31945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:07:42.066247 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31856
2026-08-06T02:07:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1495 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:42 dist-test-slave-gp6n krb5kdc[31964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:42 dist-test-slave-gp6n krb5kdc[31964](info): set up 2 sockets
Aug 06 02:07:42 dist-test-slave-gp6n krb5kdc[31964](info): commencing operation
krb5kdc: starting...
W20260806 02:07:43.129921 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:43.262333 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37633
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:43.373274 31978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:43.373585 31978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:43.373660 31978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:43.377671 31978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:43.377769 31978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:43.377810 31978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:43.377847 31978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:43.377880 31978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:43.383169 31978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:37633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:43.384289 31978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:43.385352 31978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:07:43.394157 31978 server_base.cc:1034] running on GCE node
W20260806 02:07:43.394121 31984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:43.394192 31985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:43.394121 31987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:43.394969 31978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:43.396075 31978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:43.397272 31978 hybrid_clock.cc:648] HybridClock initialized: now 1785982063397255 us; error 31 us; skew 500 ppm
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:43.400146 31978 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:43.401103 31978 webserver.cc:460] Webserver started at http://127.28.21.62:41221/ using document root <none> and password file <none>
I20260806 02:07:43.401669 31978 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:43.401763 31978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:43.402014 31978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:43.404150 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "40b82d5341944f7899b857dbe9b145dd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "7cc8a702ca4a9ca8c08b1917de56d7fe"
server_key_iv: "6c04488bf0da1819e9ad286bc13cee5c"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.404732 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "40b82d5341944f7899b857dbe9b145dd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "7cc8a702ca4a9ca8c08b1917de56d7fe"
server_key_iv: "6c04488bf0da1819e9ad286bc13cee5c"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.408699 31978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:07:43.411234 31995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.412523 31978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:43.412647 31978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "40b82d5341944f7899b857dbe9b145dd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "7cc8a702ca4a9ca8c08b1917de56d7fe"
server_key_iv: "6c04488bf0da1819e9ad286bc13cee5c"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.412758 31978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:43.426460 31978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:43.429742 31978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:43.429960 31978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:43.436661 31978 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:37633
I20260806 02:07:43.436980 32052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:37633 every 8 connection(s)
I20260806 02:07:43.437738 31978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:43.438663 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 31978
I20260806 02:07:43.438779 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:43.439118 28756 external_mini_cluster.cc:1428] Setting key 56e28d28e060b682eaa1333df47cfdd4
I20260806 02:07:43.441116 32053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:43.447467 32053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: Bootstrap starting.
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:43.449950 32053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:43.450836 32053 log.cc:824] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:43.453262 32053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: No bootstrap required, opened a new log
I20260806 02:07:43.453951 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:43.440480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.28.21.62:37633)
0806 02:07:43.440867 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:43.440891 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:43.440925 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:07:43.441600 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:43.441618 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:43.441932 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:43.442201 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:43.443140 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.444162 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.445302 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.445778 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.445897 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:43.448534 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:43.448538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:43.448551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:43.448577 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:43.450341 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.450951 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.450956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.450969 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.451151 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.451491 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.451496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.451498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.452004 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:43.452173 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:43.452849 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 02:07:43.453141 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260806 02:07:43.457201 32053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } }
I20260806 02:07:43.457471 32053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:43.457553 32053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b82d5341944f7899b857dbe9b145dd, State: Initialized, Role: FOLLOWER
I20260806 02:07:43.457978 32053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } }
I20260806 02:07:43.458130 32053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:43.458215 32053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:43.458335 32053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:43.459251 32053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } }
I20260806 02:07:43.459590 32053 leader_election.cc:302] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b82d5341944f7899b857dbe9b145dd; no voters:
I20260806 02:07:43.459908 32053 leader_election.cc:288] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:43.460067 32059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:43.460309 32059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [term 1 LEADER]: Becoming Leader. State: Replica: 40b82d5341944f7899b857dbe9b145dd, State: Running, Role: LEADER
I20260806 02:07:43.460690 32059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } }
I20260806 02:07:43.461156 32053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:43.462495 32060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b82d5341944f7899b857dbe9b145dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } } }
I20260806 02:07:43.462636 32060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:43.463195 32061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b82d5341944f7899b857dbe9b145dd. Latest consensus state: current_term: 1 leader_uuid: "40b82d5341944f7899b857dbe9b145dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b82d5341944f7899b857dbe9b145dd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37633 } } }
I20260806 02:07:43.463349 32061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd [sys.catalog]: This master's current role is: LEADER
W20260806 02:07:43.465960 32075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:43.466061 32075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:43.466169 32068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:43.466944 32068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:43.472342 32068 catalog_manager.cc:1347] Generated new cluster ID: f7ec58879f5141e19acdbbc1dab6ee54
I20260806 02:07:43.472409 32068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:43.499970 32068 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:43.501276 32068 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:43.513855 32068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b82d5341944f7899b857dbe9b145dd: Generated new TSK 0
I20260806 02:07:43.514736 32068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:43.555699 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:43.669535 32081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:43.669802 32081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:43.669894 32081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:43.674098 32081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:43.674165 32081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:43.674224 32081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:43.679612 32081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:43.680707 32081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:43.682022 32081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:43.691886 32086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:43.691883 32089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:43.691903 32087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:43.692260 32081 server_base.cc:1034] running on GCE node
I20260806 02:07:43.692713 32081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:43.693413 32081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:43.694582 32081 hybrid_clock.cc:648] HybridClock initialized: now 1785982063694540 us; error 59 us; skew 500 ppm
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:43.697645 32081 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:43.698665 32081 webserver.cc:460] Webserver started at http://127.28.21.1:45727/ using document root <none> and password file <none>
I20260806 02:07:43.699250 32081 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:43.699314 32081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:43.699536 32081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:43.701579 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "cf4b6de2503145598f1c4b1d1e8858fd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "ef458e04bf6c3212f785a3340d046460"
server_key_iv: "48c45f881661c988814bfe08dc229c41"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.702216 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "cf4b6de2503145598f1c4b1d1e8858fd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "ef458e04bf6c3212f785a3340d046460"
server_key_iv: "48c45f881661c988814bfe08dc229c41"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.706017 32081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:07:43.708479 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.709751 32081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:43.709888 32081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "cf4b6de2503145598f1c4b1d1e8858fd"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "ef458e04bf6c3212f785a3340d046460"
server_key_iv: "48c45f881661c988814bfe08dc229c41"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.710001 32081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:43.737402 32081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:43.740602 32081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:43.740808 32081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:43.741410 32081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:43.742225 32081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:43.742296 32081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.742367 32081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:43.742434 32081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.751453 32081 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:36367
I20260806 02:07:43.751482 32218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:36367 every 8 connection(s)
I20260806 02:07:43.752346 32081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:43.752811 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 32081
I20260806 02:07:43.752940 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:43.753288 28756 external_mini_cluster.cc:1428] Setting key c56fa42e95461838ddaf891e272e4e4a
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:43.774101 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:43.754203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:51483 (local address 127.28.21.62:37633)
0806 02:07:43.754336 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:43.754341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:43.755528 (+ 1187us) server_negotiation.cc:407] Connection header received
0806 02:07:43.757527 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:43.757532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:43.757593 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:43.757690 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:43.759135 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.759706 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.761596 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.761979 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.762061 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:43.770028 (+ 7967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:43.770032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:43.770034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:43.770060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:43.771612 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.772306 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.772309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.772310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.772393 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.772820 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.772822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.772823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.773008 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:43.773081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:43.773838 (+ 757us) server_negotiation.cc:299] Negotiation successful
0806 02:07:43.773966 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:43.775400 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:43.754769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37633 (local address 127.28.21.1:51483)
0806 02:07:43.755350 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:43.755411 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:43.757313 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:43.757855 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:43.757862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:43.758245 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:43.758968 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:43.758992 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.759849 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.759852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:43.761440 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:43.761453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.762142 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.762147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:43.762364 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:43.768318 (+ 5954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:43.768341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:43.769839 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:43.771745 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:43.771753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:43.771774 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:43.772185 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:43.772492 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:43.772497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:43.772502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:43.772716 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:43.773119 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:43.773129 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:43.773502 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 02:07:43.773736 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 02:07:43.774145 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":148,"threads_started":1}
I20260806 02:07:43.776696 32219 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37633
I20260806 02:07:43.776986 32219 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:43.777549 32219 heartbeater.cc:511] Master 127.28.21.62:37633 requested a full tablet report, sending...
I20260806 02:07:43.779223 32012 ts_manager.cc:194] Registered new tserver with Master: cf4b6de2503145598f1c4b1d1e8858fd (127.28.21.1:36367)
I20260806 02:07:43.780498 32012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:51483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:43.786194 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:43.899767 32224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:43.900063 32224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:43.900108 32224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:43.904104 32224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:43.904160 32224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:43.904220 32224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:43.909418 32224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:43.910463 32224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:43.911523 32224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:43.921830 32229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:43.921967 32224 server_base.cc:1034] running on GCE node
W20260806 02:07:43.921854 32230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:43.921986 32232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:43.922696 32224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:43.923393 32224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:43.924598 32224 hybrid_clock.cc:648] HybridClock initialized: now 1785982063924571 us; error 39 us; skew 500 ppm
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:43.927683 32224 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:43.928580 32224 webserver.cc:460] Webserver started at http://127.28.21.2:33413/ using document root <none> and password file <none>
I20260806 02:07:43.929163 32224 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:43.929256 32224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:43.929548 32224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:43.932588 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "df25d0a7cc8d4e48b893edbca9d4147b"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "82c289d87d0415a9a68bc16f5ed982b5"
server_key_iv: "95cd4a6f668699283b9fd9885a90d175"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.933415 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "df25d0a7cc8d4e48b893edbca9d4147b"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "82c289d87d0415a9a68bc16f5ed982b5"
server_key_iv: "95cd4a6f668699283b9fd9885a90d175"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.939040 32224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:07:43.942612 32239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.944586 32224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:43.944734 32224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "df25d0a7cc8d4e48b893edbca9d4147b"
format_stamp: "Formatted at 2026-08-06 02:07:43 on dist-test-slave-gp6n"
server_key: "82c289d87d0415a9a68bc16f5ed982b5"
server_key_iv: "95cd4a6f668699283b9fd9885a90d175"
server_key_version: "encryptionkey@0"
I20260806 02:07:43.944892 32224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:43.959251 32224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:43.963594 32224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:43.963816 32224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:43.964453 32224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:43.965324 32224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:43.965396 32224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.965464 32224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:43.965497 32224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:43.976694 32224 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:44837
I20260806 02:07:43.976765 32359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:44837 every 8 connection(s)
I20260806 02:07:43.977676 32224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:43.983541 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 32224
I20260806 02:07:43.983671 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:43.983973 28756 external_mini_cluster.cc:1428] Setting key a8e8a3f2572e3f838ca1eb4574f3a89f
Aug 06 02:07:43 dist-test-slave-gp6n krb5kdc[31964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:43.993544 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:43.979974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:48009 (local address 127.28.21.62:37633)
0806 02:07:43.980144 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:43.980148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:43.981140 (+ 992us) server_negotiation.cc:407] Connection header received
0806 02:07:43.982552 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:43.982556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:43.982617 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:43.982721 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:43.983985 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.984586 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.985762 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.986013 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.986120 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:43.989362 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:43.989365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:43.989367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:43.989392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:43.991097 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.991784 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.991786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.991787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.991863 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:43.992317 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:43.992320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:43.992321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:43.992516 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:43.992597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:43.993269 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 02:07:43.993420 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:07:43.994863 32362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:43.980385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37633 (local address 127.28.21.2:48009)
0806 02:07:43.980951 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:43.981012 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:43.982288 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:43.982905 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:43.982912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:43.983276 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:43.983838 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:43.983854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.984718 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.984721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:43.985633 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:43.985642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:43.986171 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:43.986175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:43.986366 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:43.987179 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:43.987208 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:43.989177 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:43.991225 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:43.991234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:43.991252 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:43.991671 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:43.991979 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:43.991985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:43.991990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:43.992206 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:43.992630 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:43.992639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:43.992993 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 02:07:43.993234 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 02:07:43.993646 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:43.996114 32360 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37633
I20260806 02:07:43.996408 32360 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:43.996944 32360 heartbeater.cc:511] Master 127.28.21.62:37633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:07:43.998227 32012 ts_manager.cc:194] Registered new tserver with Master: df25d0a7cc8d4e48b893edbca9d4147b (127.28.21.2:44837)
I20260806 02:07:43.998906 32012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:48009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:44.015852 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:44.129141 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:44.129442 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:44.129531 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:44.133579 32365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:44.133641 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:44.133738 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:44.139153 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:44.140226 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:44.141270 32365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:44.151229 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:44.151270 32373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:44.151224 32371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:44.151657 32365 server_base.cc:1034] running on GCE node
I20260806 02:07:44.152139 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:44.152791 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:44.154002 32365 hybrid_clock.cc:648] HybridClock initialized: now 1785982064153982 us; error 31 us; skew 500 ppm
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:44.156982 32365 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:44.157931 32365 webserver.cc:460] Webserver started at http://127.28.21.3:34965/ using document root <none> and password file <none>
I20260806 02:07:44.158558 32365 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:44.158651 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:44.158883 32365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:44.161218 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "21fecb66feac4358a07a2fb5d777f766"
format_stamp: "Formatted at 2026-08-06 02:07:44 on dist-test-slave-gp6n"
server_key: "0fcadc43ed81bbd246479b8ab35498c9"
server_key_iv: "88b8e1af785a5d3911e9f9369bf2625d"
server_key_version: "encryptionkey@0"
I20260806 02:07:44.161926 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "21fecb66feac4358a07a2fb5d777f766"
format_stamp: "Formatted at 2026-08-06 02:07:44 on dist-test-slave-gp6n"
server_key: "0fcadc43ed81bbd246479b8ab35498c9"
server_key_iv: "88b8e1af785a5d3911e9f9369bf2625d"
server_key_version: "encryptionkey@0"
I20260806 02:07:44.165964 32365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:07:44.168745 32380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:44.170152 32365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:44.170293 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "21fecb66feac4358a07a2fb5d777f766"
format_stamp: "Formatted at 2026-08-06 02:07:44 on dist-test-slave-gp6n"
server_key: "0fcadc43ed81bbd246479b8ab35498c9"
server_key_iv: "88b8e1af785a5d3911e9f9369bf2625d"
server_key_version: "encryptionkey@0"
I20260806 02:07:44.170469 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:44.184783 32365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:44.188776 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:44.189015 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:44.189699 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:44.190641 32365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:44.190694 32365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:44.190737 32365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:44.190793 32365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:44.200609 32365 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:40373
I20260806 02:07:44.200634 32505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:40373 every 8 connection(s)
I20260806 02:07:44.201606 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:44.203145 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 32365
I20260806 02:07:44.203302 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:44.203637 28756 external_mini_cluster.cc:1428] Setting key 25e0f669c7ab91f86c6db1a0997eb2e3
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[31964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982064, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:44.217383 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:44.203974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:55379 (local address 127.28.21.62:37633)
0806 02:07:44.204137 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:44.204144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:44.205202 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 02:07:44.206971 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:44.206974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:44.207022 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:44.207092 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:44.208275 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.208866 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.210129 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.210384 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.210530 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:44.213103 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:44.213106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:44.213108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:44.213132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:44.214817 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:44.215340 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:44.215344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:44.215345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:44.215447 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:44.215825 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:44.215827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:44.215829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:44.216154 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:44.216341 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:44.217042 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 02:07:44.217179 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:07:44.218681 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:44.204498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37633 (local address 127.28.21.3:55379)
0806 02:07:44.205052 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:44.205097 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:44.206773 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:44.207228 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:44.207235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:44.207611 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:44.208144 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:44.208159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.209001 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.209004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:44.210017 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:44.210027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.210581 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.210584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:44.210737 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:44.211284 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:44.211300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:44.212947 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:44.214938 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:44.214943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:44.214954 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:44.215209 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:44.215569 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:44.215572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:44.215574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:44.215719 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:44.216315 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:44.216324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:44.216750 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 02:07:44.217000 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:07:44.217415 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":134,"threads_started":1}
I20260806 02:07:44.220099 32506 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37633
I20260806 02:07:44.220362 32506 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:44.220877 32506 heartbeater.cc:511] Master 127.28.21.62:37633 requested a full tablet report, sending...
I20260806 02:07:44.222093 32012 ts_manager.cc:194] Registered new tserver with Master: 21fecb66feac4358a07a2fb5d777f766 (127.28.21.3:40373)
I20260806 02:07:44.223002 32012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:55379
I20260806 02:07:44.230963 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[31964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982064, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[31964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982064, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:44.260867 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:44.251244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.28.21.62:37633)
0806 02:07:44.251391 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:44.251406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:44.251644 (+ 238us) server_negotiation.cc:407] Connection header received
0806 02:07:44.251789 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:44.251793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:44.251865 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:44.251952 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:44.252900 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.253438 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.254336 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:44.254615 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:44.254722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:44.257274 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:44.257278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:44.257280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:44.257306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:44.258997 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:44.259468 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:44.259470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:44.259472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:44.259553 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:44.259797 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:44.259799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:44.259800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:44.260050 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:44.260200 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:44.260467 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:07:44.260572 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:07:44.261314 32012 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43462
I20260806 02:07:44.262319 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 32081
I20260806 02:07:44.271140 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 32224
I20260806 02:07:44.279556 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 32365
I20260806 02:07:44.286221 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 31978
2026-08-06T02:07:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2222 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[32518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[32518](info): set up 2 sockets
Aug 06 02:07:44 dist-test-slave-gp6n krb5kdc[32518](info): commencing operation
krb5kdc: starting...
W20260806 02:07:45.367588 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:45 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:45.501724 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38111
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:45.615425 32532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:45.615748 32532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:45.615819 32532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:45.619829 32532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:45.619922 32532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:45.619961 32532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:45.620002 32532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:45.620035 32532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:45.625788 32532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:45.626925 32532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:45.627976 32532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:45.636633 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:45.636644 32538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:45.636765 32532 server_base.cc:1034] running on GCE node
W20260806 02:07:45.636633 32539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:45.637385 32532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:45.638535 32532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:45.639699 32532 hybrid_clock.cc:648] HybridClock initialized: now 1785982065639681 us; error 31 us; skew 500 ppm
Aug 06 02:07:45 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:45.642634 32532 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:45.643471 32532 webserver.cc:460] Webserver started at http://127.28.21.62:33237/ using document root <none> and password file <none>
I20260806 02:07:45.644016 32532 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:45.644078 32532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:45.644304 32532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:45.646294 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "b275eb77684047f6a237f241ddd94249"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "67684f0587c23c625b6cbb5c39422c02"
server_key_iv: "6144b4b000db5f96ee4129fcfd9c213a"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.646921 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "b275eb77684047f6a237f241ddd94249"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "67684f0587c23c625b6cbb5c39422c02"
server_key_iv: "6144b4b000db5f96ee4129fcfd9c213a"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.650539 32532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:07:45.652854 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:45.654098 32532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:45.654238 32532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "b275eb77684047f6a237f241ddd94249"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "67684f0587c23c625b6cbb5c39422c02"
server_key_iv: "6144b4b000db5f96ee4129fcfd9c213a"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.654349 32532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:45.669893 32532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:45.672900 32532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:45.673101 32532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:45.680418 32532 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:38111
I20260806 02:07:45.680445 32604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:38111 every 8 connection(s)
I20260806 02:07:45.681340 32532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:45.684307 32605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:45.688809 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 32532
I20260806 02:07:45.688957 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:45.689255 28756 external_mini_cluster.cc:1428] Setting key 4d42652fade816487146917613680628
I20260806 02:07:45.690744 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249: Bootstrap starting.
I20260806 02:07:45.693135 32605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:45.693922 32605 log.cc:824] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:45.695909 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249: No bootstrap required, opened a new log
Aug 06 02:07:45 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:45.700700 32605 raft_consensus.cc:348] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } }
I20260806 02:07:45.700990 32605 raft_consensus.cc:374] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:45.701051 32605 raft_consensus.cc:729] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b275eb77684047f6a237f241ddd94249, State: Initialized, Role: FOLLOWER
I20260806 02:07:45.701574 32605 consensus_queue.cc:260] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } }
I20260806 02:07:45.701777 32605 raft_consensus.cc:388] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:45.701862 32605 raft_consensus.cc:482] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:45.701993 32605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:45.703307 32605 raft_consensus.cc:504] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } }
I20260806 02:07:45.703682 32605 leader_election.cc:302] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b275eb77684047f6a237f241ddd94249; no voters:
I20260806 02:07:45.704062 32605 leader_election.cc:288] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:45.704144 32610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:45.704370 32610 raft_consensus.cc:686] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [term 1 LEADER]: Becoming Leader. State: Replica: b275eb77684047f6a237f241ddd94249, State: Running, Role: LEADER
I20260806 02:07:45.704463 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:45.690846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.28.21.62:38111)
0806 02:07:45.691296 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:45.691314 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:45.691348 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:07:45.691972 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:45.691989 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:45.692305 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:45.692642 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:45.693511 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.694591 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.695452 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.695928 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.696110 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:45.699079 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:45.699085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:45.699098 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:45.699124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:45.700860 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:45.701462 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:45.701467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:45.701471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:45.701584 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:45.701962 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:45.701965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:45.701967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:45.702517 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:45.702678 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:45.703038 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 02:07:45.703378 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":146,"threads_started":1}
I20260806 02:07:45.704717 32610 consensus_queue.cc:237] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } }
I20260806 02:07:45.705480 32605 sys_catalog.cc:564] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:45.706719 32612 sys_catalog.cc:455] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b275eb77684047f6a237f241ddd94249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } } }
I20260806 02:07:45.706851 32612 sys_catalog.cc:458] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:45.707132 32611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b275eb77684047f6a237f241ddd94249. Latest consensus state: current_term: 1 leader_uuid: "b275eb77684047f6a237f241ddd94249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b275eb77684047f6a237f241ddd94249" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38111 } } }
I20260806 02:07:45.707311 32621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:45.707402 32611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:45.710430 32621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:45.716352 32621 catalog_manager.cc:1347] Generated new cluster ID: 1713987cd4a9406698199358c50789a9
I20260806 02:07:45.716424 32621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:45.734433 32621 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:45.735814 32621 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:45.754695 32621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b275eb77684047f6a237f241ddd94249: Generated new TSK 0
I20260806 02:07:45.755630 32621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:45.792112 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:45.904274 32635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:45.904539 32635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:45.904582 32635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:45.908818 32635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:45.908886 32635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:45.908972 32635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:45.914202 32635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:45.915298 32635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:45.916352 32635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:45.926224 32643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:45.926309 32635 server_base.cc:1034] running on GCE node
W20260806 02:07:45.926232 32641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:45.926645 32640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:45.927053 32635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:45.927727 32635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:45.928922 32635 hybrid_clock.cc:648] HybridClock initialized: now 1785982065928899 us; error 37 us; skew 500 ppm
Aug 06 02:07:45 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:45.931800 32635 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:45.932742 32635 webserver.cc:460] Webserver started at http://127.28.21.1:39265/ using document root <none> and password file <none>
I20260806 02:07:45.933314 32635 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:45.933408 32635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:45.933645 32635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:45.935768 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "20b045ab338e4bd5814a3641529e5053"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "68ac49e40cb09dc8240fcb794ae4fdb5"
server_key_iv: "ed4145951ffb41828b166178b07ba6cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.936355 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "20b045ab338e4bd5814a3641529e5053"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "68ac49e40cb09dc8240fcb794ae4fdb5"
server_key_iv: "ed4145951ffb41828b166178b07ba6cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.940155 32635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:45.942850 32651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:45.944088 32635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:07:45.944243 32635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "20b045ab338e4bd5814a3641529e5053"
format_stamp: "Formatted at 2026-08-06 02:07:45 on dist-test-slave-gp6n"
server_key: "68ac49e40cb09dc8240fcb794ae4fdb5"
server_key_iv: "ed4145951ffb41828b166178b07ba6cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:45.944360 32635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:45.968796 32635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:45.972206 32635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:45.972443 32635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:45.973125 32635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:45.974032 32635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:45.974109 32635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:45.974189 32635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:45.974238 32635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:45.984053 32635 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:41041
I20260806 02:07:45.984078 303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:41041 every 8 connection(s)
I20260806 02:07:45.984992 32635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:45.989764 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 32635
I20260806 02:07:45.989898 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:45.990221 28756 external_mini_cluster.cc:1428] Setting key 428663ce269ab7e20e25e15360ced79f
Aug 06 02:07:45 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:46.001896 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:45.986970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:35449 (local address 127.28.21.62:38111)
0806 02:07:45.987129 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:45.987134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:45.988056 (+ 922us) server_negotiation.cc:407] Connection header received
0806 02:07:45.989423 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:45.989426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:45.989475 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:45.989559 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:45.991215 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.992051 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.993856 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.994144 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.994271 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:45.997820 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:45.997823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:45.997835 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:45.997860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:45.999466 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.000129 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.000132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.000134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.000218 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.000644 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.000647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.000648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.000827 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:46.000907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:46.001631 (+ 724us) server_negotiation.cc:299] Negotiation successful
0806 02:07:46.001750 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:46.002910 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:45.987439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38111 (local address 127.28.21.1:35449)
0806 02:07:45.987898 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:45.987941 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:45.989188 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:45.989774 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:45.989786 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:45.990322 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:45.991010 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:45.991031 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.992195 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.992199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:45.993711 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:45.993726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:45.994317 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:45.994322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:45.994621 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:45.995345 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:45.995368 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:45.997640 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:45.999602 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:45.999612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:45.999630 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:45.999999 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.000328 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:46.000333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:46.000335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:46.000538 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.000942 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:46.000952 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:46.001296 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 02:07:46.001558 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 02:07:46.001909 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260806 02:07:46.004194 304 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38111
I20260806 02:07:46.004492 304 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:46.005016 304 heartbeater.cc:511] Master 127.28.21.62:38111 requested a full tablet report, sending...
I20260806 02:07:46.006861 32569 ts_manager.cc:194] Registered new tserver with Master: 20b045ab338e4bd5814a3641529e5053 (127.28.21.1:41041)
I20260806 02:07:46.008167 32569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:35449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:46.022079 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:46.134027 310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:46.134264 310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:46.134313 310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:46.138707 310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:46.138767 310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:46.138868 310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:46.144132 310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:46.145160 310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:46.146176 310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:46.155979 318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:46.155994 317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:46.156037 310 server_base.cc:1034] running on GCE node
W20260806 02:07:46.156068 320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:46.156739 310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:46.157444 310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:46.158661 310 hybrid_clock.cc:648] HybridClock initialized: now 1785982066158648 us; error 29 us; skew 500 ppm
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:46.161633 310 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:46.162629 310 webserver.cc:460] Webserver started at http://127.28.21.2:40843/ using document root <none> and password file <none>
I20260806 02:07:46.163187 310 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:46.163280 310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:46.163516 310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:46.165580 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "26136c56a27a49008ef6b90e1929b9c8"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "46777fc4585cc0d7bdb497c6bc031a8c"
server_key_iv: "65e86b566b822dacdccfe4bcd49286db"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.166148 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "26136c56a27a49008ef6b90e1929b9c8"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "46777fc4585cc0d7bdb497c6bc031a8c"
server_key_iv: "65e86b566b822dacdccfe4bcd49286db"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.170001 310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:07:46.172484 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.173653 310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:07:46.173790 310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "26136c56a27a49008ef6b90e1929b9c8"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "46777fc4585cc0d7bdb497c6bc031a8c"
server_key_iv: "65e86b566b822dacdccfe4bcd49286db"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.173899 310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:46.189918 310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:46.192625 310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:46.192828 310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:46.193406 310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:46.194192 310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:46.194264 310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.194335 310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:46.194386 310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.203399 310 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:33677
I20260806 02:07:46.203420 447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:33677 every 8 connection(s)
I20260806 02:07:46.204298 310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:46.209079 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 310
I20260806 02:07:46.209213 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:46.209585 28756 external_mini_cluster.cc:1428] Setting key 6c5d55ee7276eafd979ebdec962930a6
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:46.220675 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.206103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:36341 (local address 127.28.21.62:38111)
0806 02:07:46.206235 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:46.206240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:46.207126 (+ 886us) server_negotiation.cc:407] Connection header received
0806 02:07:46.208655 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:46.208659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:46.208717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:46.208791 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:46.210184 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.210855 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.212588 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.212844 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.212943 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.215720 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:46.215727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:46.215732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:46.215767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:46.217941 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.218689 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.218696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.218698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.218799 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.219240 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.219245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.219247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.219500 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:46.219594 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:46.220302 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 02:07:46.220434 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:46.222007 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.206543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38111 (local address 127.28.21.2:36341)
0806 02:07:46.206979 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:46.207020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:46.208437 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:46.208956 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:46.208962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:46.209470 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:46.209989 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:46.210004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.211012 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.211019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:46.212439 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:46.212454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.213003 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.213010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:46.213251 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.213861 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:46.213877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:46.215543 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:46.218111 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:46.218119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:46.218139 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:46.218558 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.218932 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:46.218935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:46.218937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:46.219117 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.219667 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:46.219674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:46.220000 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 02:07:46.220830 (+ 830us) client_negotiation.cc:240] Negotiation successful
0806 02:07:46.221197 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260806 02:07:46.223557 448 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:07:46.223935 448 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:46.224597 448 heartbeater.cc:511] Master 127.28.21.62:38111 requested a full tablet report, sending...
I20260806 02:07:46.225935 32569 ts_manager.cc:194] Registered new tserver with Master: 26136c56a27a49008ef6b90e1929b9c8 (127.28.21.2:33677)
I20260806 02:07:46.226752 32569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:36341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:46.243858 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:46.355495 453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:46.355737 453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:46.355780 453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:46.359923 453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:46.359983 453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:46.360044 453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:46.365193 453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:46.366169 453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:46.367192 453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:46.376663 461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:46.376652 460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:46.376772 453 server_base.cc:1034] running on GCE node
W20260806 02:07:46.376689 463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:46.377274 453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:46.378019 453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:46.379207 453 hybrid_clock.cc:648] HybridClock initialized: now 1785982066379190 us; error 27 us; skew 500 ppm
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:46.382273 453 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:46.383213 453 webserver.cc:460] Webserver started at http://127.28.21.3:39789/ using document root <none> and password file <none>
I20260806 02:07:46.383800 453 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:46.383896 453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:46.384138 453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:46.386226 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "35d3cc4ef61e4059b5376d71631f9c98"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "fe3b1b5ee6c47aedfdd4e31272aa445f"
server_key_iv: "e865eae997114257051f20593c44ad0d"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.386886 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "35d3cc4ef61e4059b5376d71631f9c98"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "fe3b1b5ee6c47aedfdd4e31272aa445f"
server_key_iv: "e865eae997114257051f20593c44ad0d"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.390784 453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:07:46.393333 471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.394734 453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:46.394888 453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "35d3cc4ef61e4059b5376d71631f9c98"
format_stamp: "Formatted at 2026-08-06 02:07:46 on dist-test-slave-gp6n"
server_key: "fe3b1b5ee6c47aedfdd4e31272aa445f"
server_key_iv: "e865eae997114257051f20593c44ad0d"
server_key_version: "encryptionkey@0"
I20260806 02:07:46.395015 453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:46.436091 453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:46.439848 453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:46.440073 453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:46.440734 453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:46.441658 453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:46.441733 453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.441812 453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:46.441869 453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:46.451720 453 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:36649
I20260806 02:07:46.451745 590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:36649 every 8 connection(s)
I20260806 02:07:46.452682 453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:46.461445 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 453
I20260806 02:07:46.461596 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:46.462031 28756 external_mini_cluster.cc:1428] Setting key d4113174ccee50c7d7fec93858806e75
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:46.467594 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.454566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:35805 (local address 127.28.21.62:38111)
0806 02:07:46.454740 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:46.454745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:46.455709 (+ 964us) server_negotiation.cc:407] Connection header received
0806 02:07:46.457191 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:46.457194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:46.457242 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:46.457325 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:46.458730 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.459403 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.460583 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.460800 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.460900 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.464335 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:46.464340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:46.464343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:46.464371 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:46.465672 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.466148 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.466150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.466151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.466225 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.466597 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.466600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.466601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.466786 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:46.466878 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:46.467368 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 02:07:46.467476 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:07:46.468343 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.455072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38111 (local address 127.28.21.3:35805)
0806 02:07:46.455553 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:46.455597 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:46.456987 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:46.457573 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:46.457582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:46.458031 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:46.458566 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:46.458581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.459540 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.459543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:46.460427 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:46.460436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.461040 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.461043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:46.461290 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.462075 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:46.462129 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:46.464106 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:46.465779 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:46.465783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:46.465795 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:46.466052 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.466300 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:46.466302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:46.466303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:46.466507 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:46.466875 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:46.466879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:46.467106 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 02:07:46.467253 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 02:07:46.467528 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
I20260806 02:07:46.469460 591 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38111
I20260806 02:07:46.469676 591 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:46.470135 591 heartbeater.cc:511] Master 127.28.21.62:38111 requested a full tablet report, sending...
I20260806 02:07:46.471166 32569 ts_manager.cc:194] Registered new tserver with Master: 35d3cc4ef61e4059b5376d71631f9c98 (127.28.21.3:36649)
I20260806 02:07:46.471730 32569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:35805
I20260806 02:07:46.475747 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:46.502699 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.492812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.28.21.1:41041)
0806 02:07:46.493145 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:46.493161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:46.493185 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:07:46.493235 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:46.493240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:46.493462 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:46.493579 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:46.494251 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.495134 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.495948 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.496313 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.496429 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.499012 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:46.499021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:46.499034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:46.499057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:46.500779 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.501229 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.501235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.501236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.501328 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.501625 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.501629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.501631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.501890 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:46.502145 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:46.502310 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:07:46.502502 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":142,"threads_started":1}
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[32518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:46.514770 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:46.505853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.28.21.62:38111)
0806 02:07:46.506009 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:46.506015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:46.506149 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:07:46.506238 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:46.506240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:46.506286 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:46.506391 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:46.507395 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.508025 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.508765 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:46.508984 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:46.509064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:46.511564 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:46.511567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:46.511569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:46.511591 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:46.513064 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.513497 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.513499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.513501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.513574 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:46.513813 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:46.513815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:46.513816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:46.514022 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:46.514126 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:46.514478 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:07:46.514575 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:07:46.515216 32569 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53968
I20260806 02:07:46.515954 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 32635
I20260806 02:07:46.525475 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 310
I20260806 02:07:46.533079 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 453
I20260806 02:07:46.538727 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 32532
2026-08-06T02:07:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2250 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[605](info): set up 2 sockets
Aug 06 02:07:46 dist-test-slave-gp6n krb5kdc[605](info): commencing operation
krb5kdc: starting...
W20260806 02:07:47.637225 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:47 dist-test-slave-gp6n krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:47.766180 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34575
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:47.875788 620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:47.876088 620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:47.876163 620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:47.880141 620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:47.880229 620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:47.880266 620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:47.880318 620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:47.880357 620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:47.885447 620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:47.886490 620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:47.887490 620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:47.895747 629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:47.895735 625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:47.895792 620 server_base.cc:1034] running on GCE node
W20260806 02:07:47.895750 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:47.896652 620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:47.897719 620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:47.898904 620 hybrid_clock.cc:648] HybridClock initialized: now 1785982067898886 us; error 30 us; skew 500 ppm
Aug 06 02:07:47 dist-test-slave-gp6n krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982067, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:47.907478 620 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:47.908172 620 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:47.908234 620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:47.908399 620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:47.910379 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "f54299c0d96b43beb1332b0ab640590c"
format_stamp: "Formatted at 2026-08-06 02:07:47 on dist-test-slave-gp6n"
server_key: "24c2f63c4d55c9398316bda9a6735a39"
server_key_iv: "cf12806e76a772889dfb0795f130f4cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:47.910969 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "f54299c0d96b43beb1332b0ab640590c"
format_stamp: "Formatted at 2026-08-06 02:07:47 on dist-test-slave-gp6n"
server_key: "24c2f63c4d55c9398316bda9a6735a39"
server_key_iv: "cf12806e76a772889dfb0795f130f4cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:47.914547 620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:47.916942 636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:47.918185 620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:47.918311 620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "f54299c0d96b43beb1332b0ab640590c"
format_stamp: "Formatted at 2026-08-06 02:07:47 on dist-test-slave-gp6n"
server_key: "24c2f63c4d55c9398316bda9a6735a39"
server_key_iv: "cf12806e76a772889dfb0795f130f4cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:47.918483 620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:47.931586 620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:47.934669 620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:47.934866 620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:47.941885 620 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:34575
I20260806 02:07:47.942242 690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:34575 every 8 connection(s)
I20260806 02:07:47.942591 620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:47.945304 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:47.950902 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: Bootstrap starting.
I20260806 02:07:47.952437 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 620
I20260806 02:07:47.952582 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:47.952848 28756 external_mini_cluster.cc:1428] Setting key 0ee8dc16677fe313a93c97838c597013
I20260806 02:07:47.953334 691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:47.954270 691 log.cc:824] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:47.956354 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: No bootstrap required, opened a new log
I20260806 02:07:47.960608 691 raft_consensus.cc:348] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } }
I20260806 02:07:47.960939 691 raft_consensus.cc:374] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:47.960989 691 raft_consensus.cc:729] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54299c0d96b43beb1332b0ab640590c, State: Initialized, Role: FOLLOWER
I20260806 02:07:47.961467 691 consensus_queue.cc:260] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } }
I20260806 02:07:47.961629 691 raft_consensus.cc:388] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:47.961719 691 raft_consensus.cc:482] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:47.961838 691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:07:47 dist-test-slave-gp6n krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:47.962975 691 raft_consensus.cc:504] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } }
I20260806 02:07:47.963388 691 leader_election.cc:302] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f54299c0d96b43beb1332b0ab640590c; no voters:
I20260806 02:07:47.963761 691 leader_election.cc:288] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:47.964035 696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:47.964319 696 raft_consensus.cc:686] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [term 1 LEADER]: Becoming Leader. State: Replica: f54299c0d96b43beb1332b0ab640590c, State: Running, Role: LEADER
I20260806 02:07:47.964648 696 consensus_queue.cc:237] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } }
I20260806 02:07:47.965137 691 sys_catalog.cc:564] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:47.966550 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f54299c0d96b43beb1332b0ab640590c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } } }
I20260806 02:07:47.966550 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f54299c0d96b43beb1332b0ab640590c. Latest consensus state: current_term: 1 leader_uuid: "f54299c0d96b43beb1332b0ab640590c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54299c0d96b43beb1332b0ab640590c" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34575 } } }
I20260806 02:07:47.966707 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:47.966707 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:47.968500 706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:47.969092 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:47.953998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.28.21.62:34575)
0806 02:07:47.954428 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:47.954450 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:47.954491 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:07:47.955112 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:47.955129 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:47.955439 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:47.955707 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:47.956635 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:47.957621 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:47.958871 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:47.959356 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:47.959481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:47.962729 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:47.962736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:47.962753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:47.962791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:47.964849 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:47.965349 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:47.965364 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:47.965370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:47.965518 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:47.966020 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:47.966025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:47.966028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:47.967121 (+ 1093us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:47.967331 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:47.967584 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 02:07:47.967866 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260806 02:07:47.971385 706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:47.976749 706 catalog_manager.cc:1347] Generated new cluster ID: 6128061354e5472ab998ffc6bbd752e6
I20260806 02:07:47.976814 706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:47.996371 706 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:47.997597 706 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:48.003602 706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: Generated new TSK 0
I20260806 02:07:48.004390 706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:48.037782 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:48.150848 717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:48.151146 717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:48.151225 717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:48.155233 717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:48.155328 717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:48.155422 717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:48.160740 717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:48.161847 717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:48.162897 717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:48.172524 722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:48.172590 723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:48.172605 725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:48.172745 717 server_base.cc:1034] running on GCE node
I20260806 02:07:48.173259 717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:48.173890 717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:48.175073 717 hybrid_clock.cc:648] HybridClock initialized: now 1785982068175057 us; error 27 us; skew 500 ppm
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:48.177788 717 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:48.178562 717 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:48.178658 717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:48.178886 717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:48.180881 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "4aec772948334396929ebbeba5434058"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "91ab810e71d0c839c7268f1e43f7b61d"
server_key_iv: "9a85f3eeef4aa175451532e77d1275d7"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.181457 717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "4aec772948334396929ebbeba5434058"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "91ab810e71d0c839c7268f1e43f7b61d"
server_key_iv: "9a85f3eeef4aa175451532e77d1275d7"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.185110 717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:07:48.187784 731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.189071 717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:48.189239 717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "4aec772948334396929ebbeba5434058"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "91ab810e71d0c839c7268f1e43f7b61d"
server_key_iv: "9a85f3eeef4aa175451532e77d1275d7"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.189397 717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:48.209525 717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:48.212739 717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:48.212965 717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:48.213593 717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:48.214596 717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:48.214660 717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.214715 717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:48.214744 717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.226560 717 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:38139
I20260806 02:07:48.226584 851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:38139 every 8 connection(s)
I20260806 02:07:48.227187 717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:48.234915 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 717
I20260806 02:07:48.235061 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:48.235338 28756 external_mini_cluster.cc:1428] Setting key bb81ab245bfae213ed0ca53469dd9c37
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:48.241616 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.228951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:55475 (local address 127.28.21.62:34575)
0806 02:07:48.229116 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.229120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.229979 (+ 859us) server_negotiation.cc:407] Connection header received
0806 02:07:48.231465 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.231469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.231535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.231633 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:48.232790 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.233342 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.234542 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.234783 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.234901 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.237663 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:48.237667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:48.237670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:48.237693 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:48.239488 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.239972 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.239978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.239983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.240075 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.240392 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.240398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.240403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.240658 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:48.240765 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.241234 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.241397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:07:48.242311 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.229319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:34575 (local address 127.28.21.1:55475)
0806 02:07:48.229805 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.229847 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.231208 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.231806 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.231813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.232176 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:48.232644 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.232656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.233479 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.233482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.234365 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.234374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.234946 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.234950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.235196 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.235846 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:48.235881 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:48.237509 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:48.239612 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.239616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.239627 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.239886 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.240168 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.240171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.240172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.240299 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.240761 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:48.240766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:48.240988 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.241138 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.241378 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":139,"threads_started":1}
I20260806 02:07:48.243611 852 heartbeater.cc:347] Connected to a master server at 127.28.21.62:34575
I20260806 02:07:48.243865 852 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:48.244406 852 heartbeater.cc:511] Master 127.28.21.62:34575 requested a full tablet report, sending...
I20260806 02:07:48.245956 651 ts_manager.cc:194] Registered new tserver with Master: 4aec772948334396929ebbeba5434058 (127.28.21.1:38139)
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:48.247237 651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:55475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:48.265775 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:48.377264 857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:48.377583 857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:48.377667 857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:48.381774 857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:48.381871 857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:48.381968 857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:48.387449 857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:48.388563 857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:48.389640 857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:07:48.399411 857 server_base.cc:1034] running on GCE node
W20260806 02:07:48.399350 863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:48.399350 862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:48.399597 865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:48.400044 857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:48.400662 857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:48.401837 857 hybrid_clock.cc:648] HybridClock initialized: now 1785982068401798 us; error 44 us; skew 500 ppm
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:48.404764 857 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:48.405537 857 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:48.405608 857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:48.405827 857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:48.407998 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "831c13292cb5418ea9b58933513b0926"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "31b795da987a84e1c9efc3c9c5b492df"
server_key_iv: "3925dbef6e6f916dd2b9e7af2917f8f3"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.408582 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "831c13292cb5418ea9b58933513b0926"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "31b795da987a84e1c9efc3c9c5b492df"
server_key_iv: "3925dbef6e6f916dd2b9e7af2917f8f3"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.412391 857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:48.414785 871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.416093 857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:48.416247 857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "831c13292cb5418ea9b58933513b0926"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "31b795da987a84e1c9efc3c9c5b492df"
server_key_iv: "3925dbef6e6f916dd2b9e7af2917f8f3"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.416364 857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:48.434870 857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:48.438385 857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:48.438642 857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:48.439247 857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:48.440093 857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:48.440162 857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.440228 857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:48.440275 857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.449075 857 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:37741
I20260806 02:07:48.449100 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:37741 every 8 connection(s)
I20260806 02:07:48.449672 857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:48.452468 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 857
I20260806 02:07:48.452605 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:48.452924 28756 external_mini_cluster.cc:1428] Setting key 1b9dbff0b250aecbe3c5e9e3ef9eb8f5
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:48.463905 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.451461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:42655 (local address 127.28.21.62:34575)
0806 02:07:48.451600 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.451604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.452474 (+ 870us) server_negotiation.cc:407] Connection header received
0806 02:07:48.454076 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.454080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.454134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.454219 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:48.455529 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.456076 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.457216 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.457458 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.457566 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.460142 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:48.460146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:48.460149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:48.460179 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:48.461741 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.462274 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.462278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.462280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.462363 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.462786 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.462791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.462792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.462985 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:48.463090 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.463597 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.463728 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:07:48.464836 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.451854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:34575 (local address 127.28.21.2:42655)
0806 02:07:48.452304 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.452347 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.453857 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.454419 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.454444 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.454803 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:48.455391 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.455407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.456202 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.456210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.457091 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.457100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.457605 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.457609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.457782 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.458388 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:48.458445 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:48.459982 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:48.461861 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.461867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.461885 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.462165 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.462501 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.462504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.462506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.462686 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.463089 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:48.463097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:48.463362 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.463557 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.463932 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:07:48.466203 998 heartbeater.cc:347] Connected to a master server at 127.28.21.62:34575
I20260806 02:07:48.466528 998 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:48.467116 998 heartbeater.cc:511] Master 127.28.21.62:34575 requested a full tablet report, sending...
I20260806 02:07:48.468304 651 ts_manager.cc:194] Registered new tserver with Master: 831c13292cb5418ea9b58933513b0926 (127.28.21.2:37741)
I20260806 02:07:48.469173 651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:42655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:48.482966 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:48.594182 1003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:48.594491 1003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:48.594542 1003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:48.598723 1003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:48.598781 1003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:48.598841 1003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:48.603893 1003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:34575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:48.604807 1003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:48.605726 1003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:48.614966 1011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:48.614946 1009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:48.615052 1003 server_base.cc:1034] running on GCE node
W20260806 02:07:48.614935 1008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:48.615577 1003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:48.616279 1003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:48.617479 1003 hybrid_clock.cc:648] HybridClock initialized: now 1785982068617473 us; error 41 us; skew 500 ppm
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:48.620512 1003 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:48.621263 1003 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:48.621336 1003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:48.621611 1003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:48.624138 1003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "7a0a84280e574595aaa6031e800ac14c"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "cdc96b4eabb773d88ce7599ef2b9fa93"
server_key_iv: "0690e8bdc23b01109e834f64cb071e85"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.624760 1003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "7a0a84280e574595aaa6031e800ac14c"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "cdc96b4eabb773d88ce7599ef2b9fa93"
server_key_iv: "0690e8bdc23b01109e834f64cb071e85"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.628686 1003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:48.631239 1018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.632510 1003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:48.632642 1003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "7a0a84280e574595aaa6031e800ac14c"
format_stamp: "Formatted at 2026-08-06 02:07:48 on dist-test-slave-gp6n"
server_key: "cdc96b4eabb773d88ce7599ef2b9fa93"
server_key_iv: "0690e8bdc23b01109e834f64cb071e85"
server_key_version: "encryptionkey@0"
I20260806 02:07:48.632757 1003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:48.672216 1003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:48.675261 1003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:48.675478 1003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:48.676083 1003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:48.676944 1003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:48.677016 1003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.677093 1003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:48.677138 1003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:48.688391 1003 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:41023
I20260806 02:07:48.688438 1138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:41023 every 8 connection(s)
I20260806 02:07:48.689069 1003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:48.689851 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1003
I20260806 02:07:48.689988 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:48.690342 28756 external_mini_cluster.cc:1428] Setting key e7e34164819d59f2a6cd73b4d893d0b9
Aug 06 02:07:48 dist-test-slave-gp6n krb5kdc[605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:48.703971 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.691001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:36121 (local address 127.28.21.62:34575)
0806 02:07:48.691279 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.691284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.692171 (+ 887us) server_negotiation.cc:407] Connection header received
0806 02:07:48.693719 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.693722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.693777 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.693854 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:48.695110 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.695653 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.696816 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.697066 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.697175 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.699962 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:48.699966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:48.699968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:48.700006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:48.701593 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.702162 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.702167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.702168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.702259 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.702678 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.702682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.702683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.702892 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:48.703005 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.703653 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.703779 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260806 02:07:48.705027 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.691542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:34575 (local address 127.28.21.3:36121)
0806 02:07:48.692032 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.692076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.693484 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.694031 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.694039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.694389 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:48.694965 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.694980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.695780 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.695784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.696688 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.696696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.697227 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.697231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.697412 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.698085 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:48.698102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:48.699786 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:48.701733 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.701739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.701754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.702049 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.702373 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:48.702380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:48.702381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:48.702576 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:48.703003 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:48.703010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:48.703322 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.703586 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.703927 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
I20260806 02:07:48.706195 1139 heartbeater.cc:347] Connected to a master server at 127.28.21.62:34575
I20260806 02:07:48.706542 1139 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:48.707031 1139 heartbeater.cc:511] Master 127.28.21.62:34575 requested a full tablet report, sending...
I20260806 02:07:48.708148 651 ts_manager.cc:194] Registered new tserver with Master: 7a0a84280e574595aaa6031e800ac14c (127.28.21.3:41023)
I20260806 02:07:48.708732 651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:36121
I20260806 02:07:48.716598 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:07:48.726668 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.718640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.28.21.62:34575)
0806 02:07:48.718797 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.718802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.718958 (+ 156us) server_negotiation.cc:407] Connection header received
0806 02:07:48.719112 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.719117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.719167 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.719252 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:48.720014 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.720603 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.721588 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.721814 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.721937 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.723341 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:48.723350 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:48.723352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:48.723375 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:48.724846 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.725293 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.725297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.725299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.725438 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:48.725731 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:48.725735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:48.725737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:48.725964 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:48.726112 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.726362 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.726499 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:07:48.729700 651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:48.732918 651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:48.747323 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.742188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.28.21.2:37741)
0806 02:07:48.742723 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.742735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.743008 (+ 273us) server_negotiation.cc:407] Connection header received
0806 02:07:48.743130 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.743135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.743325 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.743463 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:48.743676 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.744563 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.746051 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.746671 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.746795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.746948 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.747073 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.747165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":128,"threads_started":1}
I20260806 02:07:48.747316 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.741721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37741 (local address 127.0.0.1:49548)
0806 02:07:48.742883 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.742898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.743004 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.743451 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.743454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.743471 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:48.743580 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.743586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.744690 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.744695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.745919 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.745928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.746794 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.746798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.746906 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.746941 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.747048 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.747169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":104,"threads_started":1}
I20260806 02:07:48.748093 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.741462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:38139 (local address 127.0.0.1:58922)
0806 02:07:48.742273 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.742322 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.742488 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.743258 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.743265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.743289 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:48.743451 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.743458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.744906 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.744910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.746228 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.746239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.747642 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.747646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.747753 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.747777 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.747857 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.747932 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260806 02:07:48.748641 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.741670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.28.21.1:38139)
0806 02:07:48.742683 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.742691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.742718 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:07:48.742810 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.742818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.743074 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.743271 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:48.743595 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.744732 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.746388 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.747313 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.747434 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.747568 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.748368 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.748484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":83,"threads_started":1}
I20260806 02:07:48.749096 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.741844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41023 (local address 127.0.0.1:54452)
0806 02:07:48.743148 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.743161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.743248 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.743894 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.743897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.743915 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:48.744018 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.744024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.745570 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.745574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.746860 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.746871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.748489 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.748493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.748601 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.748740 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.748843 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.748934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":98,"threads_started":1}
I20260806 02:07:48.749555 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.742257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.28.21.3:41023)
0806 02:07:48.742992 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.743005 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.743339 (+ 334us) server_negotiation.cc:407] Connection header received
0806 02:07:48.743432 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.743437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.743604 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.743739 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:48.744156 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.745405 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.747012 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.747908 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.748017 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.748158 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.749153 (+ 995us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.749384 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":187,"threads_started":1}
I20260806 02:07:48.750715 925 tablet_service.cc:1467] Processing CreateTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 (DEFAULT_TABLE table=test-table [id=310c52d8a78d488f8e16bfffc5dbd908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:48.751686 925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a8a3908f834d7ca5b2acc39f6fa811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:48.752045 1069 tablet_service.cc:1467] Processing CreateTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 (DEFAULT_TABLE table=test-table [id=310c52d8a78d488f8e16bfffc5dbd908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:48.753167 1069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a8a3908f834d7ca5b2acc39f6fa811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:48.753944 779 tablet_service.cc:1467] Processing CreateTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 (DEFAULT_TABLE table=test-table [id=310c52d8a78d488f8e16bfffc5dbd908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:48.754997 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a8a3908f834d7ca5b2acc39f6fa811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:48.758044 1157 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Bootstrap starting.
I20260806 02:07:48.760175 1157 tablet_bootstrap.cc:654] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:48.760673 1158 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Bootstrap starting.
I20260806 02:07:48.761543 1157 log.cc:824] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:48.762161 1159 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: Bootstrap starting.
I20260806 02:07:48.762899 1158 tablet_bootstrap.cc:654] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:48.764045 1158 log.cc:824] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:48.764566 1159 tablet_bootstrap.cc:654] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:48.764587 1157 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: No bootstrap required, opened a new log
I20260806 02:07:48.764793 1157 ts_tablet_manager.cc:1397] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:07:48.765534 1159 log.cc:824] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:48.766059 1158 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: No bootstrap required, opened a new log
I20260806 02:07:48.766258 1158 ts_tablet_manager.cc:1397] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:07:48.767601 1159 tablet_bootstrap.cc:492] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: No bootstrap required, opened a new log
I20260806 02:07:48.767777 1159 ts_tablet_manager.cc:1397] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:07:48.768662 1157 raft_consensus.cc:348] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.768981 1157 raft_consensus.cc:374] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:48.769059 1157 raft_consensus.cc:729] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831c13292cb5418ea9b58933513b0926, State: Initialized, Role: FOLLOWER
I20260806 02:07:48.769819 1157 consensus_queue.cc:260] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.770192 1158 raft_consensus.cc:348] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.770475 1158 raft_consensus.cc:374] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:48.770540 1158 raft_consensus.cc:729] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a84280e574595aaa6031e800ac14c, State: Initialized, Role: FOLLOWER
I20260806 02:07:48.770547 998 heartbeater.cc:503] Master 127.28.21.62:34575 was elected leader, sending a full tablet report...
I20260806 02:07:48.770947 1158 consensus_queue.cc:260] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.771595 1139 heartbeater.cc:503] Master 127.28.21.62:34575 was elected leader, sending a full tablet report...
I20260806 02:07:48.772043 1157 ts_tablet_manager.cc:1428] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:07:48.772166 1159 raft_consensus.cc:348] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.772408 1159 raft_consensus.cc:374] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:48.772647 1159 raft_consensus.cc:729] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aec772948334396929ebbeba5434058, State: Initialized, Role: FOLLOWER
I20260806 02:07:48.773137 1159 consensus_queue.cc:260] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.773343 1158 ts_tablet_manager.cc:1428] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:07:48.774087 1159 ts_tablet_manager.cc:1428] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:07:48.774348 852 heartbeater.cc:503] Master 127.28.21.62:34575 was elected leader, sending a full tablet report...
I20260806 02:07:48.888751 1166 raft_consensus.cc:482] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:48.888983 1166 raft_consensus.cc:504] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.890605 1166 leader_election.cc:288] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 831c13292cb5418ea9b58933513b0926 (127.28.21.2:37741), 7a0a84280e574595aaa6031e800ac14c (127.28.21.3:41023)
I20260806 02:07:48.895483 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.890874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:48829 (local address 127.28.21.2:37741)
0806 02:07:48.891019 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.891024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.891315 (+ 291us) server_negotiation.cc:407] Connection header received
0806 02:07:48.891599 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.891604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.891668 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.891763 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:48.892030 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.892916 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.894352 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.894856 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.894944 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.894997 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.895261 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.895347 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:07:48.895519 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.890817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37741 (local address 127.28.21.1:48829)
0806 02:07:48.891190 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.891205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.891346 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.891765 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.891767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.891788 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:48.891912 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.891919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.893094 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.893097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.894215 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.894225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.894990 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.894998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.895117 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.895143 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.895230 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.895294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":5,"spinlock_wait_cycles":32000,"thread_start_us":159,"threads_started":1}
I20260806 02:07:48.895665 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.890862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41023 (local address 127.28.21.1:34843)
0806 02:07:48.891190 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.891205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:48.891346 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:48.891690 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:48.891692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:48.891711 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:48.891916 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.891920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.893068 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.893077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.894235 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:48.894245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.895111 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.895114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:48.895226 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.895251 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:07:48.895381 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 02:07:48.895450 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":79,"spinlock_wait_cycles":7808,"thread_start_us":134,"threads_started":1}
I20260806 02:07:48.896049 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.891078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:34843 (local address 127.28.21.3:41023)
0806 02:07:48.891225 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.891229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.891302 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:07:48.891476 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.891479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.891551 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.891648 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:48.892047 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.892911 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.894471 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.894972 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.895091 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.895153 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.895395 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.895472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:07:48.896333 949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a8a3908f834d7ca5b2acc39f6fa811" candidate_uuid: "4aec772948334396929ebbeba5434058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "831c13292cb5418ea9b58933513b0926" is_pre_election: true
I20260806 02:07:48.896659 949 raft_consensus.cc:2454] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aec772948334396929ebbeba5434058 in term 0.
I20260806 02:07:48.896680 1091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a8a3908f834d7ca5b2acc39f6fa811" candidate_uuid: "4aec772948334396929ebbeba5434058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a84280e574595aaa6031e800ac14c" is_pre_election: true
I20260806 02:07:48.896981 1091 raft_consensus.cc:2454] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aec772948334396929ebbeba5434058 in term 0.
I20260806 02:07:48.897202 734 leader_election.cc:302] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aec772948334396929ebbeba5434058, 831c13292cb5418ea9b58933513b0926; no voters:
I20260806 02:07:48.897524 1166 raft_consensus.cc:2781] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:48.897639 1166 raft_consensus.cc:482] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:48.897701 1166 raft_consensus.cc:3037] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:48.898846 1166 raft_consensus.cc:504] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.899215 1166 leader_election.cc:288] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [CANDIDATE]: Term 1 election: Requested vote from peers 831c13292cb5418ea9b58933513b0926 (127.28.21.2:37741), 7a0a84280e574595aaa6031e800ac14c (127.28.21.3:41023)
I20260806 02:07:48.899601 949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a8a3908f834d7ca5b2acc39f6fa811" candidate_uuid: "4aec772948334396929ebbeba5434058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "831c13292cb5418ea9b58933513b0926"
I20260806 02:07:48.899601 1091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a8a3908f834d7ca5b2acc39f6fa811" candidate_uuid: "4aec772948334396929ebbeba5434058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0a84280e574595aaa6031e800ac14c"
I20260806 02:07:48.899735 949 raft_consensus.cc:3037] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:48.899720 1091 raft_consensus.cc:3037] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:48.900815 949 raft_consensus.cc:2454] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aec772948334396929ebbeba5434058 in term 1.
I20260806 02:07:48.901010 1091 raft_consensus.cc:2454] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aec772948334396929ebbeba5434058 in term 1.
I20260806 02:07:48.901141 734 leader_election.cc:302] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aec772948334396929ebbeba5434058, 831c13292cb5418ea9b58933513b0926; no voters:
I20260806 02:07:48.901362 1166 raft_consensus.cc:2781] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:48.901651 1166 raft_consensus.cc:686] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 1 LEADER]: Becoming Leader. State: Replica: 4aec772948334396929ebbeba5434058, State: Running, Role: LEADER
I20260806 02:07:48.902254 1166 consensus_queue.cc:237] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } }
I20260806 02:07:48.905646 650 catalog_manager.cc:5495] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aec772948334396929ebbeba5434058 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "4aec772948334396929ebbeba5434058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aec772948334396929ebbeba5434058" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 38139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:48.935750 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.932251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.28.21.1:38139)
0806 02:07:48.932402 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:48.932405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:48.932419 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:07:48.932510 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:48.932513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:48.932569 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:48.932657 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:48.932917 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.933402 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.934263 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:48.934513 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:48.934591 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:48.934847 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:48.935353 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:48.935472 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:48.935530 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 02:07:48.935578 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 02:07:48.940021 1140 tablet.cc:2369] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:48.941951 1091 raft_consensus.cc:1261] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 1 FOLLOWER]: Refusing update from remote peer 4aec772948334396929ebbeba5434058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:48.941951 949 raft_consensus.cc:1261] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 1 FOLLOWER]: Refusing update from remote peer 4aec772948334396929ebbeba5434058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:48.942720 1166 consensus_queue.cc:1035] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0a84280e574595aaa6031e800ac14c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:48.942974 1170 consensus_queue.cc:1035] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "831c13292cb5418ea9b58933513b0926" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:48.950688 1173 mvcc.cc:205] Tried to move back new op lower bound from 7315382554378072064 to 7315382554226503680. Current Snapshot: MvccSnapshot[applied={T|T < 7315382554378072064}]
W20260806 02:07:48.950894 999 tablet.cc:2369] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:48.952220 1175 mvcc.cc:205] Tried to move back new op lower bound from 7315382554378072064 to 7315382554226503680. Current Snapshot: MvccSnapshot[applied={T|T < 7315382554378072064}]
I20260806 02:07:48.961253 650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:48.961406 650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:48.963928 650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f54299c0d96b43beb1332b0ab640590c: Sending DeleteTablet for 3 replicas of tablet b5a8a3908f834d7ca5b2acc39f6fa811
I20260806 02:07:48.964748 925 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:48 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:49548
I20260806 02:07:48.964864 1069 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:48 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54452
I20260806 02:07:48.964959 779 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a8a3908f834d7ca5b2acc39f6fa811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:48 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:58922
I20260806 02:07:48.965480 1183 tablet_replica.cc:330] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058: stopping tablet replica
I20260806 02:07:48.965828 1184 tablet_replica.cc:330] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: stopping tablet replica
I20260806 02:07:48.965871 1183 raft_consensus.cc:2229] T b5a8a3908f834d7ca5b2acc39f6fa811 P 4aec772948334396929ebbeba5434058 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:48.966138 1184 raft_consensus.cc:2229] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:48.966158 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 717
I20260806 02:07:48.966490 1182 tablet_replica.cc:330] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: stopping tablet replica
I20260806 02:07:48.966789 1182 raft_consensus.cc:2229] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:48.967640 1184 raft_consensus.cc:2258] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:48.968037 1182 raft_consensus.cc:2258] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:48.970007 1184 ts_tablet_manager.cc:1905] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:48.970172 1182 ts_tablet_manager.cc:1905] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:48.972239 1184 ts_tablet_manager.cc:1918] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:48.972357 1184 log.cc:1196] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/b5a8a3908f834d7ca5b2acc39f6fa811
I20260806 02:07:48.972654 1184 ts_tablet_manager.cc:1939] T b5a8a3908f834d7ca5b2acc39f6fa811 P 7a0a84280e574595aaa6031e800ac14c: Deleting consensus metadata
I20260806 02:07:48.973538 1182 ts_tablet_manager.cc:1918] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:48.973636 1182 log.cc:1196] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/b5a8a3908f834d7ca5b2acc39f6fa811
I20260806 02:07:48.973753 640 catalog_manager.cc:4841] TS 7a0a84280e574595aaa6031e800ac14c (127.28.21.3:41023): tablet b5a8a3908f834d7ca5b2acc39f6fa811 (table test-table [id=310c52d8a78d488f8e16bfffc5dbd908]) successfully deleted
I20260806 02:07:48.973958 1182 ts_tablet_manager.cc:1939] T b5a8a3908f834d7ca5b2acc39f6fa811 P 831c13292cb5418ea9b58933513b0926: Deleting consensus metadata
I20260806 02:07:48.975188 639 catalog_manager.cc:4841] TS 831c13292cb5418ea9b58933513b0926 (127.28.21.2:37741): tablet b5a8a3908f834d7ca5b2acc39f6fa811 (table test-table [id=310c52d8a78d488f8e16bfffc5dbd908]) successfully deleted
W20260806 02:07:48.975509 638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:38139 (error 108)
I20260806 02:07:48.975723 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 857
I20260806 02:07:48.978427 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:48.977975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:38139 (local address 127.0.0.1:58944)
0806 02:07:48.978208 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:48.978298 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:38139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260806 02:07:48.982971 638 catalog_manager.cc:4584] TS 4aec772948334396929ebbeba5434058 (127.28.21.1:38139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5a8a3908f834d7ca5b2acc39f6fa811: Network error: Client connection negotiation failed: client connection to 127.28.21.1:38139: connect: Connection refused (error 111)
I20260806 02:07:48.983147 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1003
I20260806 02:07:48.989588 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 620
2026-08-06T02:07:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2452 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:49Z Disabled control of system clock
I20260806 02:07:49.022249 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32863
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:07:49.134732 1190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:49.134999 1190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:49.135036 1190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:49.139185 1190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:49.139242 1190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:49.139258 1190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:49.139276 1190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:49.139292 1190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:49.144037 1190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:32863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:49.145102 1190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:49.146708 1190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:49.155524 1196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.155514 1195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.155519 1198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:49.155915 1190 server_base.cc:1034] running on GCE node
I20260806 02:07:49.156550 1190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:49.157786 1190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:49.158982 1190 hybrid_clock.cc:648] HybridClock initialized: now 1785982069158945 us; error 53 us; skew 500 ppm
I20260806 02:07:49.160987 1190 webserver.cc:460] Webserver started at http://127.28.21.62:33839/ using document root <none> and password file <none>
I20260806 02:07:49.161589 1190 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:49.161652 1190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:49.161902 1190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:49.164075 1190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "97cd656501b54def84a1f63ec628162e"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "3f1f946e9f787eb0883bc30bb7961a69"
server_key_iv: "4edaed7be6dc7de7c61141c479211bdf"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.164659 1190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "97cd656501b54def84a1f63ec628162e"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "3f1f946e9f787eb0883bc30bb7961a69"
server_key_iv: "4edaed7be6dc7de7c61141c479211bdf"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.168421 1190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:07:49.171032 1204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.172276 1190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:49.172420 1190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "97cd656501b54def84a1f63ec628162e"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "3f1f946e9f787eb0883bc30bb7961a69"
server_key_iv: "4edaed7be6dc7de7c61141c479211bdf"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.172556 1190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:49.177152 1190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:49.177831 1190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:49.178014 1190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:49.184695 1190 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:32863
I20260806 02:07:49.184712 1257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:32863 every 8 connection(s)
I20260806 02:07:49.185593 1190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:49.188066 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1190
I20260806 02:07:49.188163 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:49.188448 28756 external_mini_cluster.cc:1428] Setting key 1535be44b552549aa211e9219dbc3043
I20260806 02:07:49.188895 1258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:49.194610 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.189882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.28.21.62:32863)
0806 02:07:49.190587 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.190606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.190646 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:07:49.191640 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.191664 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.192107 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.192469 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.193152 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.193158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.193172 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.193182 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.193244 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.193264 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.193395 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.193464 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.193695 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260806 02:07:49.196122 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: Bootstrap starting.
I20260806 02:07:49.198228 1258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:49.198958 1258 log.cc:824] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:49.200649 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: No bootstrap required, opened a new log
I20260806 02:07:49.204249 1258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } }
I20260806 02:07:49.204459 1258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:49.204494 1258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cd656501b54def84a1f63ec628162e, State: Initialized, Role: FOLLOWER
I20260806 02:07:49.204869 1258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } }
I20260806 02:07:49.205039 1258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:49.205117 1258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:49.205241 1258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:49.206244 1258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } }
I20260806 02:07:49.206612 1258 leader_election.cc:302] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97cd656501b54def84a1f63ec628162e; no voters:
I20260806 02:07:49.206913 1258 leader_election.cc:288] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:49.207021 1263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:49.207273 1263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [term 1 LEADER]: Becoming Leader. State: Replica: 97cd656501b54def84a1f63ec628162e, State: Running, Role: LEADER
I20260806 02:07:49.207566 1263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } }
I20260806 02:07:49.208124 1258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:49.209308 1264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97cd656501b54def84a1f63ec628162e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } } }
I20260806 02:07:49.209456 1264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:49.209682 1265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cd656501b54def84a1f63ec628162e. Latest consensus state: current_term: 1 leader_uuid: "97cd656501b54def84a1f63ec628162e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cd656501b54def84a1f63ec628162e" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32863 } } }
I20260806 02:07:49.209786 1265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:49.209911 1272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:49.212678 1272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:49.217883 1272 catalog_manager.cc:1347] Generated new cluster ID: d6714c1c973a48be9cb43158c92089d0
I20260806 02:07:49.217958 1272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:49.235737 1272 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:49.236088 1272 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:49.241618 1272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: Generated new TSK 0
I20260806 02:07:49.242352 1272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:07:49.250083 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--builtin_ntp_servers=127.28.21.20:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:07:49.362896 1284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:49.363144 1284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:49.363184 1284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:49.367233 1284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:49.367283 1284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:49.367373 1284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:49.372015 1284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:49.373073 1284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:49.374469 1284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:49.384372 1289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.384361 1290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:49.384469 1284 server_base.cc:1034] running on GCE node
W20260806 02:07:49.384488 1292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:49.385070 1284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:49.385730 1284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:49.386919 1284 hybrid_clock.cc:648] HybridClock initialized: now 1785982069386901 us; error 27 us; skew 500 ppm
I20260806 02:07:49.388757 1284 webserver.cc:460] Webserver started at http://127.28.21.1:44751/ using document root <none> and password file <none>
I20260806 02:07:49.389349 1284 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:49.389484 1284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:49.389714 1284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:49.391865 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "c6e4bd26c2c6478f903a907c91ce3668"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "4525e1b2869447b51791aa2d8e464224"
server_key_iv: "c82e2bc46ff474ab86ffb812c71be4c3"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.392469 1284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "c6e4bd26c2c6478f903a907c91ce3668"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "4525e1b2869447b51791aa2d8e464224"
server_key_iv: "c82e2bc46ff474ab86ffb812c71be4c3"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.396183 1284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:07:49.398751 1298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.399927 1284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:07:49.400022 1284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "c6e4bd26c2c6478f903a907c91ce3668"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "4525e1b2869447b51791aa2d8e464224"
server_key_iv: "c82e2bc46ff474ab86ffb812c71be4c3"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.400106 1284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:49.404456 1284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:49.404997 1284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:49.405131 1284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:49.405603 1284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:49.406502 1284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:49.406558 1284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.406594 1284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:49.406615 1284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.415545 1284 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:36623
I20260806 02:07:49.415594 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:36623 every 8 connection(s)
I20260806 02:07:49.416507 1284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:49.416865 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1284
I20260806 02:07:49.417019 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:49.417330 28756 external_mini_cluster.cc:1428] Setting key 6f0fcb98acbe6d9f3dbb8007a46c680e
I20260806 02:07:49.419445 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--builtin_ntp_servers=127.28.21.20:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:07:49.423782 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.418428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:60725 (local address 127.28.21.62:32863)
0806 02:07:49.418571 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.418575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.420024 (+ 1449us) server_negotiation.cc:407] Connection header received
0806 02:07:49.421921 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.421926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.421971 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.422078 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.423227 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.423234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.423240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.423259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.423284 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.423299 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.423399 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.423553 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.423601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 02:07:49.424650 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.419066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32863 (local address 127.28.21.1:60725)
0806 02:07:49.419854 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.419914 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.421704 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.422220 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.422226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.422311 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.422965 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.422966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.423006 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.423072 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.423396 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.423403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.423407 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.423550 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.423803 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260806 02:07:49.425880 1419 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32863
I20260806 02:07:49.426090 1419 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:49.426542 1419 heartbeater.cc:511] Master 127.28.21.62:32863 requested a full tablet report, sending...
I20260806 02:07:49.428190 1221 ts_manager.cc:194] Registered new tserver with Master: c6e4bd26c2c6478f903a907c91ce3668 (127.28.21.1:36623)
W20260806 02:07:49.530778 1422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:49.531038 1422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:49.531077 1422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:49.534984 1422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:49.535030 1422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:49.535082 1422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:49.539620 1422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:49.540593 1422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:49.541898 1422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:49.551257 1431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.551261 1427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.551589 1428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:49.551721 1422 server_base.cc:1034] running on GCE node
I20260806 02:07:49.552148 1422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:49.552775 1422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:49.553967 1422 hybrid_clock.cc:648] HybridClock initialized: now 1785982069553946 us; error 32 us; skew 500 ppm
I20260806 02:07:49.555832 1422 webserver.cc:460] Webserver started at http://127.28.21.2:34363/ using document root <none> and password file <none>
I20260806 02:07:49.556476 1422 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:49.556571 1422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:49.556802 1422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:49.558897 1422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "c10cd362659547cf8abd87e63548e8d1"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7558390e7013e4bc1d7f4d6d402267e4"
server_key_iv: "a7a7407fb299b6da2460deb460b1ac38"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.559466 1422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "c10cd362659547cf8abd87e63548e8d1"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7558390e7013e4bc1d7f4d6d402267e4"
server_key_iv: "a7a7407fb299b6da2460deb460b1ac38"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.563005 1422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:49.565397 1439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.566633 1422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:49.566773 1422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "c10cd362659547cf8abd87e63548e8d1"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7558390e7013e4bc1d7f4d6d402267e4"
server_key_iv: "a7a7407fb299b6da2460deb460b1ac38"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.566885 1422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:49.571369 1422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:49.571986 1422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:49.572171 1422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:49.572695 1422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:49.573571 1422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:49.573643 1422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.573714 1422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:49.573758 1422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.582486 1422 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:46327
I20260806 02:07:49.582533 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:46327 every 8 connection(s)
I20260806 02:07:49.583362 1422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:49.586100 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1422
I20260806 02:07:49.586228 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:49.586602 28756 external_mini_cluster.cc:1428] Setting key 5f7213245a39ce96375567476a084dce
I20260806 02:07:49.588693 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--builtin_ntp_servers=127.28.21.20:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:07:49.589843 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.585119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:36807 (local address 127.28.21.62:32863)
0806 02:07:49.585258 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.585262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.586506 (+ 1244us) server_negotiation.cc:407] Connection header received
0806 02:07:49.588112 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.588115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.588154 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.588229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.589381 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.589385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.589388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.589401 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.589425 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.589438 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.589522 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.589665 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.589709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:07:49.590775 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.585503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32863 (local address 127.28.21.2:36807)
0806 02:07:49.586270 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.586371 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.587879 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.588404 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.588411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.588508 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.589114 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.589116 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.589155 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.589222 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.589509 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.589517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.589522 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.589696 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.589941 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260806 02:07:49.591931 1559 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32863
I20260806 02:07:49.592146 1559 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:49.592525 1559 heartbeater.cc:511] Master 127.28.21.62:32863 requested a full tablet report, sending...
I20260806 02:07:49.593550 1221 ts_manager.cc:194] Registered new tserver with Master: c10cd362659547cf8abd87e63548e8d1 (127.28.21.2:46327)
W20260806 02:07:49.700886 1562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:49.701151 1562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:49.701191 1562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:49.705260 1562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:49.705317 1562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:49.705374 1562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:49.710181 1562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:49.711249 1562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:49.712581 1562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:49.721982 1573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.722023 1568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:49.722024 1569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:49.722275 1562 server_base.cc:1034] running on GCE node
I20260806 02:07:49.722693 1562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:49.723379 1562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:49.724567 1562 hybrid_clock.cc:648] HybridClock initialized: now 1785982069724538 us; error 42 us; skew 500 ppm
I20260806 02:07:49.726365 1562 webserver.cc:460] Webserver started at http://127.28.21.3:36779/ using document root <none> and password file <none>
I20260806 02:07:49.726951 1562 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:49.727044 1562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:49.727272 1562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:49.729247 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "caecd9af94ab4840bd9aa31d03bb220d"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7efab7972543ac7469cfdc39effc6546"
server_key_iv: "d667b6eb25775f0793625ca57e2dbb23"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.729820 1562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "caecd9af94ab4840bd9aa31d03bb220d"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7efab7972543ac7469cfdc39effc6546"
server_key_iv: "d667b6eb25775f0793625ca57e2dbb23"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.733409 1562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 02:07:49.735918 1579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.737102 1562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 02:07:49.737228 1562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "caecd9af94ab4840bd9aa31d03bb220d"
format_stamp: "Formatted at 2026-08-06 02:07:49 on dist-test-slave-gp6n"
server_key: "7efab7972543ac7469cfdc39effc6546"
server_key_iv: "d667b6eb25775f0793625ca57e2dbb23"
server_key_version: "encryptionkey@0"
I20260806 02:07:49.737336 1562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:49.741714 1562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:49.742276 1562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:49.742501 1562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:49.743052 1562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:49.743891 1562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:49.743944 1562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.743980 1562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:49.744000 1562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:49.753134 1562 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:40339
I20260806 02:07:49.753162 1698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:40339 every 8 connection(s)
I20260806 02:07:49.753963 1562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:49.755723 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1562
I20260806 02:07:49.755860 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:49.756137 28756 external_mini_cluster.cc:1428] Setting key 54d09dbd0f69865e43e5f613c5d64f6c
I20260806 02:07:49.760493 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.755712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:46793 (local address 127.28.21.62:32863)
0806 02:07:49.755880 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.755888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.757126 (+ 1238us) server_negotiation.cc:407] Connection header received
0806 02:07:49.758715 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.758718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.758754 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.758829 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.759989 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.759995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.759999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.760017 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.760053 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.760067 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.760178 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.760313 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.760360 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:07:49.761278 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.756203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32863 (local address 127.28.21.3:46793)
0806 02:07:49.756977 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.757029 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.758530 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.758950 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.758956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.759037 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.759645 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.759650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.759705 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.759791 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.760168 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.760177 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.760182 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.760348 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.760585 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260806 02:07:49.762326 1699 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32863
I20260806 02:07:49.762550 1699 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:49.762905 1699 heartbeater.cc:511] Master 127.28.21.62:32863 requested a full tablet report, sending...
I20260806 02:07:49.763801 1221 ts_manager.cc:194] Registered new tserver with Master: caecd9af94ab4840bd9aa31d03bb220d (127.28.21.3:40339)
I20260806 02:07:49.769615 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:07:49.773125 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.771510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.28.21.62:32863)
0806 02:07:49.771645 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.771650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.771824 (+ 174us) server_negotiation.cc:407] Connection header received
0806 02:07:49.771972 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.771975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.772008 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.772095 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.772710 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.772713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.772716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.772727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.772745 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.772753 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.772847 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.772963 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.773005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:07:49.775687 1221 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:49.778846 1221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:49.790910 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.787583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:40339 (local address 127.0.0.1:34088)
0806 02:07:49.788458 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.788474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.788571 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.789147 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.789154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.789236 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.789956 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.789959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.789984 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.790061 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.790527 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.790539 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.790545 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.790671 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.790745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":105,"threads_started":1}
I20260806 02:07:49.791000 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.788247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.28.21.3:40339)
0806 02:07:49.788624 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.788635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.788657 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:07:49.788709 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.788713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.788910 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.789055 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.790229 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.790239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.790254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.790276 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.790355 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.790398 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.790546 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.790702 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.790785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":162,"threads_started":1}
I20260806 02:07:49.791117 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.787596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.28.21.2:46327)
0806 02:07:49.788360 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.788373 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.788650 (+ 277us) server_negotiation.cc:407] Connection header received
0806 02:07:49.789085 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.789091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.789268 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.789418 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.790497 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.790506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.790520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.790534 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.790576 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.790613 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.790692 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.790867 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.790934 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":113,"threads_started":1}
I20260806 02:07:49.791303 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.788139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36623 (local address 127.0.0.1:57782)
0806 02:07:49.788911 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.788925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.788999 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.790056 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.790060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.790076 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.790472 (+ 396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.790477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.790492 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.790523 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.791012 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.791019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.791027 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.791126 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.791168 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":96,"threads_started":1}
I20260806 02:07:49.791666 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.788301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.28.21.1:36623)
0806 02:07:49.789026 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.789041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.789080 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:07:49.789138 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.789144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.789403 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.789578 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.790695 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.790701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.790713 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.790726 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.790771 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.790802 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.790956 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.791413 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.791487 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":137,"threads_started":1}
I20260806 02:07:49.791482 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.787448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:46327 (local address 127.0.0.1:47370)
0806 02:07:49.788007 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.788063 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.788715 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.789813 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.789816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.789831 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.790257 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.790260 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.790271 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.790299 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.790759 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.790762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.790764 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.791300 (+ 536us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.791337 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":78,"threads_started":1}
I20260806 02:07:49.794483 1348 tablet_service.cc:1467] Processing CreateTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba (DEFAULT_TABLE table=test-table [id=c38e4dffe5064f9eb657a525a0000695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:49.794536 1489 tablet_service.cc:1467] Processing CreateTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba (DEFAULT_TABLE table=test-table [id=c38e4dffe5064f9eb657a525a0000695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:49.794540 1631 tablet_service.cc:1467] Processing CreateTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba (DEFAULT_TABLE table=test-table [id=c38e4dffe5064f9eb657a525a0000695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:49.795580 1631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:49.795580 1489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:49.795594 1348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:49.802125 1717 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Bootstrap starting.
I20260806 02:07:49.803198 1718 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Bootstrap starting.
I20260806 02:07:49.803725 1719 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Bootstrap starting.
I20260806 02:07:49.805056 1717 tablet_bootstrap.cc:654] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:49.805401 1718 tablet_bootstrap.cc:654] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:49.805982 1719 tablet_bootstrap.cc:654] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:49.806037 1717 log.cc:824] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:49.806113 1718 log.cc:824] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:49.806885 1719 log.cc:824] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:49.808152 1717 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: No bootstrap required, opened a new log
I20260806 02:07:49.808346 1718 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: No bootstrap required, opened a new log
I20260806 02:07:49.808364 1717 ts_tablet_manager.cc:1397] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:07:49.808549 1718 ts_tablet_manager.cc:1397] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:49.809024 1719 tablet_bootstrap.cc:492] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: No bootstrap required, opened a new log
I20260806 02:07:49.809212 1719 ts_tablet_manager.cc:1397] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:07:49.812310 1717 raft_consensus.cc:348] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.812563 1717 raft_consensus.cc:374] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:49.812618 1717 raft_consensus.cc:729] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e4bd26c2c6478f903a907c91ce3668, State: Initialized, Role: FOLLOWER
I20260806 02:07:49.813063 1717 consensus_queue.cc:260] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.813683 1419 heartbeater.cc:503] Master 127.28.21.62:32863 was elected leader, sending a full tablet report...
I20260806 02:07:49.813583 1718 raft_consensus.cc:348] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.813855 1718 raft_consensus.cc:374] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:49.813925 1718 raft_consensus.cc:729] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10cd362659547cf8abd87e63548e8d1, State: Initialized, Role: FOLLOWER
I20260806 02:07:49.814065 1717 ts_tablet_manager.cc:1428] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:07:49.814172 1719 raft_consensus.cc:348] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.814476 1719 raft_consensus.cc:374] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:49.814440 1718 consensus_queue.cc:260] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.814549 1719 raft_consensus.cc:729] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caecd9af94ab4840bd9aa31d03bb220d, State: Initialized, Role: FOLLOWER
I20260806 02:07:49.815006 1719 consensus_queue.cc:260] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.815547 1718 ts_tablet_manager.cc:1428] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 02:07:49.815689 1699 heartbeater.cc:503] Master 127.28.21.62:32863 was elected leader, sending a full tablet report...
I20260806 02:07:49.815878 1559 heartbeater.cc:503] Master 127.28.21.62:32863 was elected leader, sending a full tablet report...
I20260806 02:07:49.817045 1719 ts_tablet_manager.cc:1428] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
W20260806 02:07:49.834296 1560 tablet.cc:2369] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:49.880827 1724 raft_consensus.cc:482] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:49.881088 1724 raft_consensus.cc:504] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.882300 1724 leader_election.cc:288] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caecd9af94ab4840bd9aa31d03bb220d (127.28.21.3:40339), c6e4bd26c2c6478f903a907c91ce3668 (127.28.21.1:36623)
I20260806 02:07:49.884406 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.882602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36623 (local address 127.28.21.2:52135)
0806 02:07:49.882718 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.882735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.882830 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.883178 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.883183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.883198 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.883751 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.883752 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.883763 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.883790 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.884132 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.884135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.884139 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.884228 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.884273 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 02:07:49.884429 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.882723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:52135 (local address 127.28.21.1:36623)
0806 02:07:49.882902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.882909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.882924 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:07:49.882988 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.882993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.883039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.883127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.883920 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.883929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.883931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.883961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.883993 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.884006 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.884104 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.884253 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.884300 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:07:49.884938 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.882867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:51195 (local address 127.28.21.3:40339)
0806 02:07:49.883008 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.883012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.883434 (+ 422us) server_negotiation.cc:407] Connection header received
0806 02:07:49.883568 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.883572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.883608 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.883705 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.884467 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.884473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.884477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.884490 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.884513 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.884527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.884605 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.884769 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.884810 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:49.885270 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.882602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:40339 (local address 127.28.21.2:51195)
0806 02:07:49.883296 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.883313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:49.883460 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:49.883734 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:49.883737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:49.883754 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:49.884282 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:07:49.884284 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:49.884294 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:07:49.884317 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:49.884680 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:49.884682 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:49.884685 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:07:49.885100 (+ 415us) client_negotiation.cc:240] Negotiation successful
0806 02:07:49.885136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":298,"thread_start_us":143,"threads_started":1}
I20260806 02:07:49.885215 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff1f681f8e643a2bd3dc8ae2dc8f3ba" candidate_uuid: "c10cd362659547cf8abd87e63548e8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e4bd26c2c6478f903a907c91ce3668" is_pre_election: true
I20260806 02:07:49.885533 1369 raft_consensus.cc:2454] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c10cd362659547cf8abd87e63548e8d1 in term 0.
I20260806 02:07:49.885941 1651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff1f681f8e643a2bd3dc8ae2dc8f3ba" candidate_uuid: "c10cd362659547cf8abd87e63548e8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caecd9af94ab4840bd9aa31d03bb220d" is_pre_election: true
I20260806 02:07:49.885990 1442 leader_election.cc:302] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c10cd362659547cf8abd87e63548e8d1, c6e4bd26c2c6478f903a907c91ce3668; no voters:
I20260806 02:07:49.886221 1651 raft_consensus.cc:2454] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c10cd362659547cf8abd87e63548e8d1 in term 0.
I20260806 02:07:49.886257 1724 raft_consensus.cc:2781] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:49.886338 1724 raft_consensus.cc:482] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:49.886430 1724 raft_consensus.cc:3037] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:49.887532 1724 raft_consensus.cc:504] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.887898 1724 leader_election.cc:288] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [CANDIDATE]: Term 1 election: Requested vote from peers caecd9af94ab4840bd9aa31d03bb220d (127.28.21.3:40339), c6e4bd26c2c6478f903a907c91ce3668 (127.28.21.1:36623)
I20260806 02:07:49.888170 1651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff1f681f8e643a2bd3dc8ae2dc8f3ba" candidate_uuid: "c10cd362659547cf8abd87e63548e8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caecd9af94ab4840bd9aa31d03bb220d"
I20260806 02:07:49.888222 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff1f681f8e643a2bd3dc8ae2dc8f3ba" candidate_uuid: "c10cd362659547cf8abd87e63548e8d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e4bd26c2c6478f903a907c91ce3668"
I20260806 02:07:49.888299 1651 raft_consensus.cc:3037] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:49.888348 1369 raft_consensus.cc:3037] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:49.889353 1651 raft_consensus.cc:2454] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c10cd362659547cf8abd87e63548e8d1 in term 1.
I20260806 02:07:49.889391 1369 raft_consensus.cc:2454] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c10cd362659547cf8abd87e63548e8d1 in term 1.
I20260806 02:07:49.889647 1442 leader_election.cc:302] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c10cd362659547cf8abd87e63548e8d1, c6e4bd26c2c6478f903a907c91ce3668; no voters:
I20260806 02:07:49.889834 1724 raft_consensus.cc:2781] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:49.890106 1724 raft_consensus.cc:686] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 1 LEADER]: Becoming Leader. State: Replica: c10cd362659547cf8abd87e63548e8d1, State: Running, Role: LEADER
I20260806 02:07:49.890471 1724 consensus_queue.cc:237] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } }
I20260806 02:07:49.893713 1221 catalog_manager.cc:5495] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c10cd362659547cf8abd87e63548e8d1 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "c10cd362659547cf8abd87e63548e8d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cd362659547cf8abd87e63548e8d1" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 46327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 } health_report { overall_health: UNKNOWN } } }
I20260806 02:07:49.915299 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:49.913597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.28.21.2:46327)
0806 02:07:49.913750 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:49.913754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:49.913768 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:07:49.913940 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:49.913943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:49.914005 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:49.914101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:49.914859 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:49.914862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:49.914865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:49.914876 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:49.914896 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:49.914908 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:49.915007 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:49.915111 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 02:07:49.915152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:07:49.918021 1420 tablet.cc:2369] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:49.921494 1369 raft_consensus.cc:1261] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c6e4bd26c2c6478f903a907c91ce3668 [term 1 FOLLOWER]: Refusing update from remote peer c10cd362659547cf8abd87e63548e8d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:49.921500 1651 raft_consensus.cc:1261] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 1 FOLLOWER]: Refusing update from remote peer c10cd362659547cf8abd87e63548e8d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:49.922228 1728 consensus_queue.cc:1035] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caecd9af94ab4840bd9aa31d03bb220d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:49.922470 1724 consensus_queue.cc:1035] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e4bd26c2c6478f903a907c91ce3668" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:49.930858 1733 mvcc.cc:205] Tried to move back new op lower bound from 7315382558390620160 to 7315382558274043904. Current Snapshot: MvccSnapshot[applied={T|T < 7315382558390620160}]
I20260806 02:07:49.939004 1221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:49.939152 1221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:49.941596 1221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97cd656501b54def84a1f63ec628162e: Sending DeleteTablet for 3 replicas of tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba
I20260806 02:07:49.942521 1631 tablet_service.cc:1514] Processing DeleteTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:49 UTC) from {username='slave'} at 127.0.0.1:34088
I20260806 02:07:49.942577 1348 tablet_service.cc:1514] Processing DeleteTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:49 UTC) from {username='slave'} at 127.0.0.1:57782
I20260806 02:07:49.943128 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1284
I20260806 02:07:49.943193 1489 tablet_service.cc:1514] Processing DeleteTablet for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:49 UTC) from {username='slave'} at 127.0.0.1:47370
I20260806 02:07:49.944888 1746 tablet_replica.cc:330] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: stopping tablet replica
I20260806 02:07:49.945204 1746 raft_consensus.cc:2229] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:49.945456 1744 tablet_replica.cc:330] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: stopping tablet replica
I20260806 02:07:49.945608 1746 raft_consensus.cc:2258] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:49.945765 1744 raft_consensus.cc:2229] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:49.946076 1744 raft_consensus.cc:2258] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:49.947091 1746 ts_tablet_manager.cc:1905] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:49.947499 1744 ts_tablet_manager.cc:1905] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:49.949940 1746 ts_tablet_manager.cc:1918] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:49.950039 1746 log.cc:1196] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/bff1f681f8e643a2bd3dc8ae2dc8f3ba
I20260806 02:07:49.950392 1746 ts_tablet_manager.cc:1939] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P c10cd362659547cf8abd87e63548e8d1: Deleting consensus metadata
I20260806 02:07:49.951275 1744 ts_tablet_manager.cc:1918] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:49.951372 1744 log.cc:1196] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/bff1f681f8e643a2bd3dc8ae2dc8f3ba
I20260806 02:07:49.951713 1744 ts_tablet_manager.cc:1939] T bff1f681f8e643a2bd3dc8ae2dc8f3ba P caecd9af94ab4840bd9aa31d03bb220d: Deleting consensus metadata
I20260806 02:07:49.951918 1206 catalog_manager.cc:4841] TS c10cd362659547cf8abd87e63548e8d1 (127.28.21.2:46327): tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba (table test-table [id=c38e4dffe5064f9eb657a525a0000695]) successfully deleted
I20260806 02:07:49.952908 1207 catalog_manager.cc:4841] TS caecd9af94ab4840bd9aa31d03bb220d (127.28.21.3:40339): tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba (table test-table [id=c38e4dffe5064f9eb657a525a0000695]) successfully deleted
W20260806 02:07:49.953531 1206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:36623 (error 108)
I20260806 02:07:49.954530 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1422
W20260806 02:07:49.954640 1714 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:49.954272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36623 (local address 127.0.0.1:57794)
0806 02:07:49.954474 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:49.954547 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:36623: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 02:07:49.957262 1206 catalog_manager.cc:4584] TS c6e4bd26c2c6478f903a907c91ce3668 (127.28.21.1:36623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bff1f681f8e643a2bd3dc8ae2dc8f3ba: Network error: Client connection negotiation failed: client connection to 127.28.21.1:36623: connect: Connection reset by peer (error 104)
I20260806 02:07:49.961272 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1562
I20260806 02:07:49.967352 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1190
2026-08-06T02:07:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (976 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:07:49.990173 28756 mini_oidc.cc:152] Starting JWKS server
I20260806 02:07:49.992038 28756 webserver.cc:460] Webserver started at https://127.0.0.1:46415/ using document root <none> and password file <none>
I20260806 02:07:49.992231 28756 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:07:49.992511 28756 webserver.cc:460] Webserver started at http://127.0.0.1:40503/ using document root <none> and password file <none>
2026-08-06T02:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:49Z Disabled control of system clock
I20260806 02:07:50.002362 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46805
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:45817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46805
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46415/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/testchainca.cert with env {}
W20260806 02:07:50.115370 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:50.115658 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:50.115701 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:50.119836 1754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:50.119902 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:50.119918 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:50.119936 1754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:50.119951 1754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:07:50.119968 1754 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:07:50.119982 1754 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46415/jwks.json
I20260806 02:07:50.125342 1754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46415/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:50.126444 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:50.127432 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:50.136330 1759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:50.136384 1760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:50.136397 1762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:50.136787 1754 server_base.cc:1034] running on GCE node
I20260806 02:07:50.137338 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:50.138579 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:50.139763 1754 hybrid_clock.cc:648] HybridClock initialized: now 1785982070139742 us; error 32 us; skew 500 ppm
I20260806 02:07:50.141779 1754 webserver.cc:460] Webserver started at http://127.28.21.62:33531/ using document root <none> and password file <none>
I20260806 02:07:50.142357 1754 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:50.142516 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:50.142719 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:50.144927 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "f29447ea2d12438a948015e55bd9d0b4"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "958712f6cfc76c93a9ef1ea9781212d2"
server_key_iv: "bdc564a2db7b2a8e3aefcd4ef1c84350"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.145485 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "f29447ea2d12438a948015e55bd9d0b4"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "958712f6cfc76c93a9ef1ea9781212d2"
server_key_iv: "bdc564a2db7b2a8e3aefcd4ef1c84350"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.149322 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:07:50.152034 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:50.153373 1754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:07:50.153479 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "f29447ea2d12438a948015e55bd9d0b4"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "958712f6cfc76c93a9ef1ea9781212d2"
server_key_iv: "bdc564a2db7b2a8e3aefcd4ef1c84350"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.153563 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:50.169559 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:50.170257 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:50.170511 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:50.178193 1754 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:46805
I20260806 02:07:50.178221 1824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:46805 every 8 connection(s)
I20260806 02:07:50.179301 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:50.182547 1825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:50.188129 1825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4: Bootstrap starting.
I20260806 02:07:50.188732 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1754
I20260806 02:07:50.188869 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:50.189145 28756 external_mini_cluster.cc:1428] Setting key bfad38dce5ed46b983c53483523838f8
I20260806 02:07:50.191125 1825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:50.191881 1825 log.cc:824] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:50.193805 1825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4: No bootstrap required, opened a new log
I20260806 02:07:50.197297 1825 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } }
I20260806 02:07:50.197538 1825 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:50.197629 1825 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29447ea2d12438a948015e55bd9d0b4, State: Initialized, Role: FOLLOWER
I20260806 02:07:50.198053 1825 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } }
I20260806 02:07:50.198203 1825 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:50.198273 1825 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:50.198372 1825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:50.199473 1825 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } }
I20260806 02:07:50.199656 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.190589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.28.21.62:46805)
0806 02:07:50.191324 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.191344 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.191381 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:07:50.192102 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.192121 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.192587 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.192992 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:50.193465 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.194679 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.195687 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.196243 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.196451 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.197385 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:50.197391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:50.197404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:50.198002 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:50.198065 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:50.198088 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:50.198207 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:50.198353 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:07:50.198669 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260806 02:07:50.199822 1825 leader_election.cc:302] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29447ea2d12438a948015e55bd9d0b4; no voters:
I20260806 02:07:50.200162 1825 leader_election.cc:288] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:50.200289 1832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:50.200579 1832 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [term 1 LEADER]: Becoming Leader. State: Replica: f29447ea2d12438a948015e55bd9d0b4, State: Running, Role: LEADER
I20260806 02:07:50.200915 1832 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } }
I20260806 02:07:50.202106 1825 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:50.202570 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:07:50.202602 1833 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29447ea2d12438a948015e55bd9d0b4. Latest consensus state: current_term: 1 leader_uuid: "f29447ea2d12438a948015e55bd9d0b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } } }
I20260806 02:07:50.202750 1833 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.203121 1834 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29447ea2d12438a948015e55bd9d0b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29447ea2d12438a948015e55bd9d0b4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46805 } } }
I20260806 02:07:50.203229 1834 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.203708 1837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:50.207677 1837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:50.213779 1837 catalog_manager.cc:1347] Generated new cluster ID: a438fda2dc1a4fa4b779d0722017cc10
I20260806 02:07:50.213879 1837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:50.256202 1837 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:50.257511 1837 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:50.265035 1837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29447ea2d12438a948015e55bd9d0b4: Generated new TSK 0
I20260806 02:07:50.265817 1837 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:07:50.292738 1860 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:07:50.292953 1860 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:07:50.293608 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.276712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.28.21.62:46805)
0806 02:07:50.276837 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.276842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.277055 (+ 213us) server_negotiation.cc:407] Connection header received
0806 02:07:50.277296 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.277298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.277379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.277487 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:07:50.277756 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.278275 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.279227 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.279463 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.279577 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.279940 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:07:50.293181 (+ 13241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:07:50.293469 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:07:50.293702 1830 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:07:50.295589 1858 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:50.276624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46805 (local address 127.0.0.1:35686)
0806 02:07:50.276968 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:50.276981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:50.277124 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:50.277506 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:50.277509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:50.277517 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:07:50.277644 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.277649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.278444 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.278448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.279101 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.279108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.279584 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.279588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.279686 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.279734 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:07:50.293391 (+ 13657us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:07:50.293621 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.21.62:46805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.21.62:46805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46415/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:07:50.296311 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1754
2026-08-06T02:07:50Z chronyd exiting
I20260806 02:07:50.335901 28756 test_util.cc:175] -----------------------------------------------
I20260806 02:07:50.335999 28756 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785982043785074-28756-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:07:50.337167 28756 mini_oidc.cc:152] Starting JWKS server
I20260806 02:07:50.338075 28756 webserver.cc:460] Webserver started at https://127.0.0.1:44123/ using document root <none> and password file <none>
I20260806 02:07:50.338184 28756 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:07:50.338588 28756 webserver.cc:460] Webserver started at http://127.0.0.1:46073/ using document root <none> and password file <none>
2026-08-06T02:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:50Z Disabled control of system clock
I20260806 02:07:50.348277 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43581
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:43581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/testchainca.cert with env {}
W20260806 02:07:50.458936 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:50.459251 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:50.459323 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:50.463399 1866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:50.463496 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:50.463526 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:50.463567 1866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:50.463604 1866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:07:50.463644 1866 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:07:50.463677 1866 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44123/jwks.json
I20260806 02:07:50.469200 1866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:43581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:50.470335 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:50.471410 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:50.479975 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:50.479957 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:50.480031 1866 server_base.cc:1034] running on GCE node
W20260806 02:07:50.479957 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:50.480785 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:50.481879 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:50.483045 1866 hybrid_clock.cc:648] HybridClock initialized: now 1785982070483032 us; error 25 us; skew 500 ppm
I20260806 02:07:50.484859 1866 webserver.cc:460] Webserver started at http://127.28.21.62:42935/ using document root <none> and password file <none>
I20260806 02:07:50.485436 1866 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:50.485500 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:50.485752 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:50.487833 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "2b23448fe97345f6a63f261d12d01ec0"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "c2d2ebd67c2e91455a6f230cae46d7e7"
server_key_iv: "b19e7dcdbc8c44e513048964d4f6b8ad"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.488396 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "2b23448fe97345f6a63f261d12d01ec0"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "c2d2ebd67c2e91455a6f230cae46d7e7"
server_key_iv: "b19e7dcdbc8c44e513048964d4f6b8ad"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.492082 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 02:07:50.494537 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:50.495790 1866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:50.495939 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "2b23448fe97345f6a63f261d12d01ec0"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "c2d2ebd67c2e91455a6f230cae46d7e7"
server_key_iv: "b19e7dcdbc8c44e513048964d4f6b8ad"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.496054 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:50.536144 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:50.536906 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:50.537124 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:50.544525 1935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:43581 every 8 connection(s)
I20260806 02:07:50.544530 1866 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:43581
I20260806 02:07:50.545548 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:50.548949 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:50.554701 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0: Bootstrap starting.
I20260806 02:07:50.554744 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1866
I20260806 02:07:50.554859 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:50.555167 28756 external_mini_cluster.cc:1428] Setting key e8f8c1fc5604bb6f70450926846cfdcd
I20260806 02:07:50.557310 1936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:50.558064 1936 log.cc:824] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:50.560118 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0: No bootstrap required, opened a new log
I20260806 02:07:50.564530 1936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } }
I20260806 02:07:50.564692 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.556506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.28.21.62:43581)
0806 02:07:50.557184 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.557198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.557233 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:07:50.557916 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.557942 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.558388 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.558806 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:50.559188 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.560157 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.561357 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.561844 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.561971 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.562889 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:50.562894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:50.562907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:50.563307 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:50.563367 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:50.563387 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:50.563470 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:50.563669 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:07:50.563969 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260806 02:07:50.564790 1936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:50.564870 1936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b23448fe97345f6a63f261d12d01ec0, State: Initialized, Role: FOLLOWER
I20260806 02:07:50.565351 1936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } }
I20260806 02:07:50.565512 1936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:50.565577 1936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:50.565668 1936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:50.566676 1936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } }
I20260806 02:07:50.567065 1936 leader_election.cc:302] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b23448fe97345f6a63f261d12d01ec0; no voters:
I20260806 02:07:50.567449 1936 leader_election.cc:288] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:50.567580 1942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:50.567768 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:07:50.567793 1942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [term 1 LEADER]: Becoming Leader. State: Replica: 2b23448fe97345f6a63f261d12d01ec0, State: Running, Role: LEADER
I20260806 02:07:50.568094 1942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } }
I20260806 02:07:50.568763 1936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:50.569229 1942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b23448fe97345f6a63f261d12d01ec0. Latest consensus state: current_term: 1 leader_uuid: "2b23448fe97345f6a63f261d12d01ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } } }
I20260806 02:07:50.569365 1942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.569644 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b23448fe97345f6a63f261d12d01ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b23448fe97345f6a63f261d12d01ec0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43581 } } }
I20260806 02:07:50.569756 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.569766 1948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:50.573029 1948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:50.578869 1948 catalog_manager.cc:1347] Generated new cluster ID: 74b09bf992184a06a70e3f0f228b16a3
I20260806 02:07:50.578954 1948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:50.606076 1948 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:50.607084 1948 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:50.615502 1948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b23448fe97345f6a63f261d12d01ec0: Generated new TSK 0
I20260806 02:07:50.616219 1948 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:07:50.652429 1970 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:07:50.652524 1970 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:07:50.653254 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.629562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.28.21.62:43581)
0806 02:07:50.629714 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.629720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.629953 (+ 233us) server_negotiation.cc:407] Connection header received
0806 02:07:50.630074 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.630077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.630123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.630217 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:07:50.630532 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.631121 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.632596 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.632824 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.632921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.633251 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:07:50.652900 (+ 19649us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:07:50.653130 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 02:07:50.653344 1940 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:07:50.653491 1969 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:50.629481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43581 (local address 127.0.0.1:48588)
0806 02:07:50.629825 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:50.629844 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:50.629962 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:50.630243 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:50.630246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:50.630255 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:07:50.630370 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.630375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.631250 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.631254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.632253 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.632264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.632962 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.632966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.633058 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.633087 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:07:50.653096 (+ 20009us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:07:50.653295 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.21.62:43581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":131,"threads_started":1}
I20260806 02:07:50.654255 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1866
2026-08-06T02:07:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (358 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:07:50.695545 28756 mini_oidc.cc:152] Starting JWKS server
I20260806 02:07:50.696465 28756 webserver.cc:460] Webserver started at https://127.0.0.1:44385/ using document root <none> and password file <none>
I20260806 02:07:50.696599 28756 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:07:50.696880 28756 webserver.cc:460] Webserver started at http://127.0.0.1:34237/ using document root <none> and password file <none>
2026-08-06T02:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:50Z Disabled control of system clock
I20260806 02:07:50.706799 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36605
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:36605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:07:50.814661 1977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:50.814939 1977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:50.814981 1977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:50.818898 1977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:50.818949 1977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:50.818965 1977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:50.818982 1977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:50.818997 1977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:07:50.819012 1977 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:07:50.819026 1977 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44385/jwks.json
I20260806 02:07:50.823633 1977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:36605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:50.824664 1977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:50.826018 1977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:50.834846 1983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:50.834867 1982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:50.834961 1977 server_base.cc:1034] running on GCE node
W20260806 02:07:50.834858 1985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:50.835734 1977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:50.836802 1977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:50.837972 1977 hybrid_clock.cc:648] HybridClock initialized: now 1785982070837942 us; error 44 us; skew 500 ppm
I20260806 02:07:50.839852 1977 webserver.cc:460] Webserver started at http://127.28.21.62:42229/ using document root <none> and password file <none>
I20260806 02:07:50.840444 1977 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:50.840507 1977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:50.840720 1977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:50.843251 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "ca633e5ba12742deb9340ba13a556904"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "d4646a5bf128d73f864c47e140744c55"
server_key_iv: "e41b8c0e1aa2e97b36cae92ff67bbb42"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.843907 1977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "ca633e5ba12742deb9340ba13a556904"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "d4646a5bf128d73f864c47e140744c55"
server_key_iv: "e41b8c0e1aa2e97b36cae92ff67bbb42"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.847720 1977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:07:50.850231 1991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:50.851643 1977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:50.851749 1977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "ca633e5ba12742deb9340ba13a556904"
format_stamp: "Formatted at 2026-08-06 02:07:50 on dist-test-slave-gp6n"
server_key: "d4646a5bf128d73f864c47e140744c55"
server_key_iv: "e41b8c0e1aa2e97b36cae92ff67bbb42"
server_key_version: "encryptionkey@0"
I20260806 02:07:50.851890 1977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:50.871065 1977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:50.871816 1977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:50.872059 1977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:50.879473 1977 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:36605
I20260806 02:07:50.879524 2050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:36605 every 8 connection(s)
I20260806 02:07:50.880493 1977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:50.882529 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 1977
I20260806 02:07:50.882673 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:50.882959 28756 external_mini_cluster.cc:1428] Setting key fe4e4071db02fd15ac666dcb6a5e667f
I20260806 02:07:50.884209 2051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:50.891583 2051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904: Bootstrap starting.
I20260806 02:07:50.894239 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.884407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.28.21.62:36605)
0806 02:07:50.885072 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.885093 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.885135 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:07:50.886005 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.886029 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.886553 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.886961 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:50.887494 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.889252 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.890183 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.891005 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.891182 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.892006 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:50.892013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:50.892030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:50.892559 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:50.892636 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:50.892667 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:50.892777 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:50.892923 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:07:50.893232 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260806 02:07:50.894587 2051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:50.895566 2051 log.cc:824] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:50.897332 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:07:50.897924 2051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904: No bootstrap required, opened a new log
I20260806 02:07:50.901540 2051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } }
I20260806 02:07:50.901818 2051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:50.901898 2051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca633e5ba12742deb9340ba13a556904, State: Initialized, Role: FOLLOWER
I20260806 02:07:50.902295 2051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } }
I20260806 02:07:50.902463 2051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:50.902551 2051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:50.902676 2051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:50.903599 2051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } }
I20260806 02:07:50.903937 2051 leader_election.cc:302] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca633e5ba12742deb9340ba13a556904; no voters:
I20260806 02:07:50.904237 2051 leader_election.cc:288] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:50.904385 2056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:50.904604 2056 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [term 1 LEADER]: Becoming Leader. State: Replica: ca633e5ba12742deb9340ba13a556904, State: Running, Role: LEADER
I20260806 02:07:50.904920 2056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } }
I20260806 02:07:50.905447 2051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:50.906570 2057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca633e5ba12742deb9340ba13a556904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } } }
I20260806 02:07:50.906698 2057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.907004 2058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca633e5ba12742deb9340ba13a556904. Latest consensus state: current_term: 1 leader_uuid: "ca633e5ba12742deb9340ba13a556904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca633e5ba12742deb9340ba13a556904" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36605 } } }
I20260806 02:07:50.907160 2058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:50.907083 2065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:50.909864 2065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:50.915112 2065 catalog_manager.cc:1347] Generated new cluster ID: a326bd657f374159b77c78d16649666d
I20260806 02:07:50.915223 2065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:50.926111 2065 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:50.927052 2065 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:50.932148 2065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca633e5ba12742deb9340ba13a556904: Generated new TSK 0
I20260806 02:07:50.932837 2065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:07:50.950105 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:50.944965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.28.21.62:36605)
0806 02:07:50.945120 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:50.945125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:50.945346 (+ 221us) server_negotiation.cc:407] Connection header received
0806 02:07:50.945496 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:50.945499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:50.945546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:50.945646 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:07:50.945930 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.946541 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.947780 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.947983 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.948051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.948372 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:07:50.949747 (+ 1375us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:07:50.949964 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:07:50.950172 2054 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:07:50.952013 2081 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:50.944905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36605 (local address 127.0.0.1:47174)
0806 02:07:50.945223 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:50.945237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:50.945358 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:50.945670 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:50.945674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:50.945682 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:07:50.945794 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.945801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.946650 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.946654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.947660 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:50.947670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:50.948089 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:50.948093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:50.948186 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:50.948214 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:07:50.949916 (+ 1702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:07:50.951816 (+ 1900us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.21.62:36605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":1698,"thread_start_us":131,"threads_started":1}
I20260806 02:07:50.952574 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 1977
E20260806 02:07:50.961726 2082 webserver.cc:533] Webserver: sslize failed:
E20260806 02:07:50.961822 2082 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T02:07:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:50Z Disabled control of system clock
I20260806 02:07:51.002750 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34303
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:34303
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/testchainca.cert with env {}
W20260806 02:07:51.112581 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:51.112836 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:51.112876 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:51.116847 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:51.116904 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:51.116920 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:51.116936 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:51.116951 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:07:51.116966 2086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:07:51.116981 2086 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:07:51.122146 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:34303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:51.123216 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:51.124183 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:51.132979 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:51.132947 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:51.133121 2086 server_base.cc:1034] running on GCE node
W20260806 02:07:51.133384 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:51.133966 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:51.135176 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:51.136371 2086 hybrid_clock.cc:648] HybridClock initialized: now 1785982071136351 us; error 33 us; skew 500 ppm
I20260806 02:07:51.138258 2086 webserver.cc:460] Webserver started at http://127.28.21.62:36127/ using document root <none> and password file <none>
I20260806 02:07:51.139019 2086 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:51.139122 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:51.139359 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:51.141408 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "966bd16950804e98972995ae86698664"
format_stamp: "Formatted at 2026-08-06 02:07:51 on dist-test-slave-gp6n"
server_key: "333940a01455d7e588ece903bdd54e62"
server_key_iv: "45484208eb5fe65c1b45482eee740ce0"
server_key_version: "encryptionkey@0"
I20260806 02:07:51.141984 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "966bd16950804e98972995ae86698664"
format_stamp: "Formatted at 2026-08-06 02:07:51 on dist-test-slave-gp6n"
server_key: "333940a01455d7e588ece903bdd54e62"
server_key_iv: "45484208eb5fe65c1b45482eee740ce0"
server_key_version: "encryptionkey@0"
I20260806 02:07:51.146014 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:07:51.148627 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:51.149840 2086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:51.150000 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "966bd16950804e98972995ae86698664"
format_stamp: "Formatted at 2026-08-06 02:07:51 on dist-test-slave-gp6n"
server_key: "333940a01455d7e588ece903bdd54e62"
server_key_iv: "45484208eb5fe65c1b45482eee740ce0"
server_key_version: "encryptionkey@0"
I20260806 02:07:51.150122 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:51.171758 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:51.172500 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:51.172709 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:51.179831 2086 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:34303
I20260806 02:07:51.179862 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:34303 every 8 connection(s)
I20260806 02:07:51.180783 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:51.183693 2157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:51.188685 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2086
I20260806 02:07:51.188834 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:51.189112 28756 external_mini_cluster.cc:1428] Setting key 19136a8a3e7ffdcfa2c6c32997ff6448
I20260806 02:07:51.189467 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664: Bootstrap starting.
I20260806 02:07:51.192178 2157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:51.193069 2157 log.cc:824] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:51.195276 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664: No bootstrap required, opened a new log
I20260806 02:07:51.199829 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:51.190580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.28.21.62:34303)
0806 02:07:51.191266 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:51.191293 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:51.191336 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:07:51.192200 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:51.192226 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:51.192743 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:51.193114 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:51.193609 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.195126 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.196054 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.196734 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.196901 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:51.197666 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:51.197672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:51.197689 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:07:51.198193 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:51.198270 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:07:51.198297 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:51.198436 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:51.198584 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:07:51.198913 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260806 02:07:51.199663 2157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } }
I20260806 02:07:51.199968 2157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:51.200028 2157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966bd16950804e98972995ae86698664, State: Initialized, Role: FOLLOWER
I20260806 02:07:51.200512 2157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } }
I20260806 02:07:51.200641 2157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:51.200723 2157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:51.200829 2157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:51.201941 2157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } }
I20260806 02:07:51.202322 2157 leader_election.cc:302] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966bd16950804e98972995ae86698664; no voters:
I20260806 02:07:51.202699 2157 leader_election.cc:288] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:51.202813 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:07:51.202821 2162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:51.203097 2162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [term 1 LEADER]: Becoming Leader. State: Replica: 966bd16950804e98972995ae86698664, State: Running, Role: LEADER
I20260806 02:07:51.203449 2162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } }
I20260806 02:07:51.204027 2157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:51.205278 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966bd16950804e98972995ae86698664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } } }
I20260806 02:07:51.205407 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:51.205766 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966bd16950804e98972995ae86698664. Latest consensus state: current_term: 1 leader_uuid: "966bd16950804e98972995ae86698664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966bd16950804e98972995ae86698664" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34303 } } }
I20260806 02:07:51.205844 2172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:51.205937 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:51.208389 2172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:51.214252 2172 catalog_manager.cc:1347] Generated new cluster ID: 67d822f1e6ce4195a8bc557298e6f758
I20260806 02:07:51.214344 2172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:51.233240 2172 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:51.234171 2172 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:51.240483 2172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 966bd16950804e98972995ae86698664: Generated new TSK 0
I20260806 02:07:51.241161 2172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:07:51.327425 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:51.253861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.28.21.62:34303)
0806 02:07:51.254108 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:51.254112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:51.254232 (+ 120us) server_negotiation.cc:407] Connection header received
0806 02:07:51.254471 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:51.254477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:51.254584 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:51.254678 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:07:51.254998 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.255559 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.256776 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.256998 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.257072 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:51.257394 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:07:51.327036 (+ 69642us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:07:51.327303 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43328: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":132}
W20260806 02:07:51.327549 2160 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43328: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:07:51.327739 2188 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:51.253796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:34303 (local address 127.0.0.1:43328)
0806 02:07:51.254133 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:51.254148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:51.254265 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:51.254699 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:51.254702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:51.254711 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:07:51.254823 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:51.254830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.255669 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.255673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:51.256666 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:51.256676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:51.257113 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:51.257117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:51.257213 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:51.257241 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:07:51.327292 (+ 70051us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:07:51.327540 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.21.62:34303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260806 02:07:51.328397 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2086
2026-08-06T02:07:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (355 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:51 dist-test-slave-gp6n krb5kdc[2200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:51 dist-test-slave-gp6n krb5kdc[2200](info): set up 2 sockets
Aug 06 02:07:51 dist-test-slave-gp6n krb5kdc[2200](info): commencing operation
krb5kdc: starting...
W20260806 02:07:52.681164 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:52 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:07:52.810663 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45655
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:52.921989 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:52.922250 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:52.922290 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:52.926348 2216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:52.926422 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:52.926445 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:52.926461 2216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:52.926477 2216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:52.931715 2216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:52.932796 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:52.933836 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:52.942519 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:52.942528 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:52.942528 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:52.942914 2216 server_base.cc:1034] running on GCE node
I20260806 02:07:52.943473 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:52.944578 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:52.945775 2216 hybrid_clock.cc:648] HybridClock initialized: now 1785982072945762 us; error 34 us; skew 500 ppm
Aug 06 02:07:52 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:52.952082 2216 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:07:52.953140 2216 webserver.cc:460] Webserver started at http://127.28.21.62:39059/ using document root <none> and password file <none>
I20260806 02:07:52.953732 2216 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:52.953828 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:52.954072 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:52.956219 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "566cc22f6453447ab428e928d2aaae3b"
format_stamp: "Formatted at 2026-08-06 02:07:52 on dist-test-slave-gp6n"
server_key: "eb7a01ac762534e9e74107867e8d031e"
server_key_iv: "578c5b655a7f1200848a25135598e2cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:52.956874 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "566cc22f6453447ab428e928d2aaae3b"
format_stamp: "Formatted at 2026-08-06 02:07:52 on dist-test-slave-gp6n"
server_key: "eb7a01ac762534e9e74107867e8d031e"
server_key_iv: "578c5b655a7f1200848a25135598e2cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:52.960852 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:07:52.963537 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:52.964898 2216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:52.965076 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "566cc22f6453447ab428e928d2aaae3b"
format_stamp: "Formatted at 2026-08-06 02:07:52 on dist-test-slave-gp6n"
server_key: "eb7a01ac762534e9e74107867e8d031e"
server_key_iv: "578c5b655a7f1200848a25135598e2cc"
server_key_version: "encryptionkey@0"
I20260806 02:07:52.965202 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:52.978477 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:52.981379 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:52.981585 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:52.988921 2216 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:45655
I20260806 02:07:52.988994 2283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:45655 every 8 connection(s)
I20260806 02:07:52.989948 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:52.992738 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:52.997144 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2216
I20260806 02:07:52.997289 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:52.997617 28756 external_mini_cluster.cc:1428] Setting key c1502b865c0f1ec3cd6b2dac54a72934
I20260806 02:07:52.998463 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b: Bootstrap starting.
I20260806 02:07:53.000954 2284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:53.001920 2284 log.cc:824] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:53.004221 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b: No bootstrap required, opened a new log
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:53.008828 2284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } }
I20260806 02:07:53.009101 2284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:53.009192 2284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566cc22f6453447ab428e928d2aaae3b, State: Initialized, Role: FOLLOWER
I20260806 02:07:53.009662 2284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } }
I20260806 02:07:53.009814 2284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:53.009922 2284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:53.010040 2284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:53.011216 2284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } }
I20260806 02:07:53.011576 2284 leader_election.cc:302] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 566cc22f6453447ab428e928d2aaae3b; no voters:
I20260806 02:07:53.011940 2284 leader_election.cc:288] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:53.012013 2289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:53.012249 2289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [term 1 LEADER]: Becoming Leader. State: Replica: 566cc22f6453447ab428e928d2aaae3b, State: Running, Role: LEADER
I20260806 02:07:53.012609 2289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } }
I20260806 02:07:53.013319 2284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:53.013316 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:52.998995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.28.21.62:45655)
0806 02:07:52.999404 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:52.999420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:52.999456 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:07:53.000093 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.000111 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.000448 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.000732 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.001719 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.002832 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.003989 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.004542 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.004673 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.007508 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.007513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.007526 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.007553 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.009757 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.010278 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.010283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.010287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.010514 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.010806 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.010809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.010810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.011256 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.011541 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.011840 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.012371 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260806 02:07:53.014313 2290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "566cc22f6453447ab428e928d2aaae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } } }
I20260806 02:07:53.014791 2290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:53.014675 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 566cc22f6453447ab428e928d2aaae3b. Latest consensus state: current_term: 1 leader_uuid: "566cc22f6453447ab428e928d2aaae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566cc22f6453447ab428e928d2aaae3b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45655 } } }
I20260806 02:07:53.015165 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:53.016027 2301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:53.019011 2301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:53.025827 2301 catalog_manager.cc:1347] Generated new cluster ID: ae9538302a304ed0b26b35b893ece05c
I20260806 02:07:53.025926 2301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:53.036957 2301 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:53.038204 2301 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:53.048098 2301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 566cc22f6453447ab428e928d2aaae3b: Generated new TSK 0
I20260806 02:07:53.048921 2301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:07:53.081770 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:53.193877 2312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:53.194157 2312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:53.194234 2312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:53.198385 2312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:53.198513 2312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:53.198604 2312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:53.203763 2312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:53.204866 2312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:53.205900 2312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:53.215550 2318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:53.215564 2317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:53.215564 2320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:53.215813 2312 server_base.cc:1034] running on GCE node
I20260806 02:07:53.216327 2312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:53.216897 2312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:53.218060 2312 hybrid_clock.cc:648] HybridClock initialized: now 1785982073218048 us; error 29 us; skew 500 ppm
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.220815 2312 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:07:53.221650 2312 webserver.cc:460] Webserver started at http://127.28.21.1:38029/ using document root <none> and password file <none>
I20260806 02:07:53.222234 2312 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:53.222340 2312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:53.222597 2312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:53.224619 2312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "fef6f835af96458a9754d309d6ad238f"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "666a7ebc1731ca3665f8e71faef7c9be"
server_key_iv: "17766a83069107019efa7f028bb66be3"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.225179 2312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "fef6f835af96458a9754d309d6ad238f"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "666a7ebc1731ca3665f8e71faef7c9be"
server_key_iv: "17766a83069107019efa7f028bb66be3"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.228965 2312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:53.231437 2329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.232710 2312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:53.232834 2312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "fef6f835af96458a9754d309d6ad238f"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "666a7ebc1731ca3665f8e71faef7c9be"
server_key_iv: "17766a83069107019efa7f028bb66be3"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.232954 2312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:53.249888 2312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:53.252826 2312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:53.252978 2312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:53.253508 2312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:53.254266 2312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:53.254311 2312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.254350 2312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:53.254366 2312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.263484 2312 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:43211
I20260806 02:07:53.263514 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:43211 every 8 connection(s)
I20260806 02:07:53.264338 2312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:53.268533 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2312
I20260806 02:07:53.268680 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:53.268955 28756 external_mini_cluster.cc:1428] Setting key 4c4054963d1be01c4fd2cd3584dde394
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:53.280486 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.266054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:44701 (local address 127.28.21.62:45655)
0806 02:07:53.266222 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.266227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.267067 (+ 840us) server_negotiation.cc:407] Connection header received
0806 02:07:53.268524 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.268527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.268572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.268664 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.270278 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.271148 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.272526 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.273003 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.273132 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.276196 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.276201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.276205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.276237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.278047 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.278641 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.278644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.278646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.278747 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.279258 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.279264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.279267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.279513 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.279622 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.280124 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.280260 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:07:53.281167 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.266457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45655 (local address 127.28.21.1:44701)
0806 02:07:53.266921 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:53.266963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:53.268220 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:53.268835 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:53.268841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:53.269255 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:53.270052 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.270075 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.271309 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.271313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.272390 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.272401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.273199 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.273204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.273437 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.274083 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:53.274109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:53.276016 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:53.278181 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.278186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.278197 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.278537 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.278852 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.278855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.278856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.278996 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.279625 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:53.279630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:53.279861 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 02:07:53.280013 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 02:07:53.280256 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260806 02:07:53.282356 2449 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45655
I20260806 02:07:53.282691 2449 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:53.283291 2449 heartbeater.cc:511] Master 127.28.21.62:45655 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:53.285019 2248 ts_manager.cc:194] Registered new tserver with Master: fef6f835af96458a9754d309d6ad238f (127.28.21.1:43211)
I20260806 02:07:53.286260 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:44701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:07:53.301411 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:53.411540 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:53.411856 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:53.411904 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:53.416273 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:53.416349 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:53.416411 2454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:53.421782 2454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:53.422851 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:53.423872 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:53.433516 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:53.433575 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:53.433681 2454 server_base.cc:1034] running on GCE node
W20260806 02:07:53.433574 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:53.434330 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:53.434985 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:53.436146 2454 hybrid_clock.cc:648] HybridClock initialized: now 1785982073436128 us; error 30 us; skew 500 ppm
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.439213 2454 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:07:53.440055 2454 webserver.cc:460] Webserver started at http://127.28.21.2:39961/ using document root <none> and password file <none>
I20260806 02:07:53.440645 2454 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:53.440711 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:53.440886 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:53.443069 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "b9bc1c1aba324fcfa80bb38ed8894adf"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "071f7ed37267ce89212bf08771db64f4"
server_key_iv: "776b2de0d62171626ca2ea416521fc6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.443619 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "b9bc1c1aba324fcfa80bb38ed8894adf"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "071f7ed37267ce89212bf08771db64f4"
server_key_iv: "776b2de0d62171626ca2ea416521fc6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.447398 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:07:53.449780 2471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.451081 2454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:07:53.451196 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "b9bc1c1aba324fcfa80bb38ed8894adf"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "071f7ed37267ce89212bf08771db64f4"
server_key_iv: "776b2de0d62171626ca2ea416521fc6b"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.451285 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:53.466899 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:53.470105 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:53.470260 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:53.470862 2454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:53.471644 2454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:53.471688 2454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.471727 2454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:53.471743 2454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.480607 2454 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:43535
I20260806 02:07:53.480635 2592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:43535 every 8 connection(s)
I20260806 02:07:53.481520 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:53.488548 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2454
I20260806 02:07:53.488699 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:53.489017 28756 external_mini_cluster.cc:1428] Setting key 2d3554f9584de4a30b01daad5bf14ede
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:53.496629 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.483459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:54709 (local address 127.28.21.62:45655)
0806 02:07:53.483622 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.483626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.484435 (+ 809us) server_negotiation.cc:407] Connection header received
0806 02:07:53.485819 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.485823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.485871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.485965 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.487173 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.487703 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.489262 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.489560 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.489675 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.492569 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.492576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.492579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.492610 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.494086 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.494748 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.494750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.494752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.494842 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.495298 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.495302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.495304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.495532 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.495614 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.496319 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.496428 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:07:53.497757 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.483830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45655 (local address 127.28.21.2:54709)
0806 02:07:53.484268 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:53.484311 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:53.485605 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:53.486094 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:53.486100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:53.486557 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:53.487041 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.487054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.487826 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.487829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.489147 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.489158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.489872 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.489875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.490021 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.490667 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:53.490683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:53.492354 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:53.494207 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.494215 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.494233 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.494645 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.494946 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.494952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.494957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.495151 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.495648 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:53.495656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:53.496000 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 02:07:53.496225 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 02:07:53.496603 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260806 02:07:53.499205 2593 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45655
I20260806 02:07:53.499621 2593 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:53.500203 2593 heartbeater.cc:511] Master 127.28.21.62:45655 requested a full tablet report, sending...
I20260806 02:07:53.501359 2248 ts_manager.cc:194] Registered new tserver with Master: b9bc1c1aba324fcfa80bb38ed8894adf (127.28.21.2:43535)
I20260806 02:07:53.501951 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:54709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:07:53.521049 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:53.633296 2598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:53.633586 2598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:53.633671 2598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:53.638099 2598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:53.638177 2598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:53.638262 2598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:53.643550 2598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:53.644593 2598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:53.645624 2598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:53.655308 2603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:53.655318 2606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:53.655432 2604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:53.655715 2598 server_base.cc:1034] running on GCE node
I20260806 02:07:53.656114 2598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:53.656708 2598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:53.657925 2598 hybrid_clock.cc:648] HybridClock initialized: now 1785982073657915 us; error 29 us; skew 500 ppm
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.661064 2598 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:07:53.662009 2598 webserver.cc:460] Webserver started at http://127.28.21.3:42911/ using document root <none> and password file <none>
I20260806 02:07:53.662715 2598 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:53.662818 2598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:53.663070 2598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:53.665232 2598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "09c5cd9458394ec69a2b578f7d418fd4"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "d62a15873f37e7cf89877a976afb14ee"
server_key_iv: "9648392c3fc923f73d0322ce677041e8"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.665865 2598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "09c5cd9458394ec69a2b578f7d418fd4"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "d62a15873f37e7cf89877a976afb14ee"
server_key_iv: "9648392c3fc923f73d0322ce677041e8"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.670120 2598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:07:53.672768 2613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.674008 2598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:07:53.674154 2598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "09c5cd9458394ec69a2b578f7d418fd4"
format_stamp: "Formatted at 2026-08-06 02:07:53 on dist-test-slave-gp6n"
server_key: "d62a15873f37e7cf89877a976afb14ee"
server_key_iv: "9648392c3fc923f73d0322ce677041e8"
server_key_version: "encryptionkey@0"
I20260806 02:07:53.674268 2598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:53.687880 2598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:53.690991 2598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:53.691193 2598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:53.691787 2598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:53.692618 2598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:53.692689 2598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.692759 2598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:53.692811 2598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:53.702069 2598 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:39221
I20260806 02:07:53.702100 2736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:39221 every 8 connection(s)
I20260806 02:07:53.702996 2598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:53.707763 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2598
I20260806 02:07:53.707934 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:53.708240 28756 external_mini_cluster.cc:1428] Setting key fc003fad151dcde5a3ad50bd40d13ec4
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:07:53.716967 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.704947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59085 (local address 127.28.21.62:45655)
0806 02:07:53.705094 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.705098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.705941 (+ 843us) server_negotiation.cc:407] Connection header received
0806 02:07:53.707477 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.707481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.707540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.707621 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.708943 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.709499 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.710708 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.710941 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.711073 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.713550 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.713553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.713555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.713581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.715071 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.715601 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.715604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.715605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.715681 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.715993 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.715995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.715996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.716174 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.716249 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.716741 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.716838 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:07:53.717875 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.705316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45655 (local address 127.28.21.3:59085)
0806 02:07:53.705765 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:53.705809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:53.707256 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:53.707788 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:53.707795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:53.708253 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:53.708784 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.708798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.709628 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.709631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.710588 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:53.710597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.711109 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.711114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:53.711288 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.711831 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:53.711847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:53.713378 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:53.715225 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.715230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.715241 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.715509 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.715771 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:53.715773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:53.715774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:53.715914 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:53.716281 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:53.716286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:53.716533 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 02:07:53.716696 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 02:07:53.716946 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260806 02:07:53.719033 2737 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45655
I20260806 02:07:53.719271 2737 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:53.719769 2737 heartbeater.cc:511] Master 127.28.21.62:45655 requested a full tablet report, sending...
I20260806 02:07:53.720808 2248 ts_manager.cc:194] Registered new tserver with Master: 09c5cd9458394ec69a2b578f7d418fd4 (127.28.21.3:39221)
I20260806 02:07:53.721364 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:59085
I20260806 02:07:53.722141 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/test-admin.keytab.
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.740106 28756 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:07:53.743111 2744 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:53.741466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.28.21.1:43211)
0806 02:07:53.741827 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.741844 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.741876 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:07:53.741956 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.741964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.742163 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.742379 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.742919 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58384: BlockingRecv error: recv got EOF from 127.0.0.1:58384 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":129,"threads_started":1}
W20260806 02:07:53.743252 28756 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.744328 28756 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.745223 28756 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:53.755376 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.746066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.28.21.1:43211)
0806 02:07:53.746202 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.746206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.746301 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:07:53.746529 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.746532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.746579 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.746668 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.747206 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.747989 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.748857 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.749239 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.749363 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.751806 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.751815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.751827 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.751850 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.753406 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.753849 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.753854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.753855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.753940 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.754278 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.754282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.754284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.754594 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.754813 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.754999 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.755160 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 02:07:53.759389 2744 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:53.758180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.28.21.1:43211)
0806 02:07:53.758294 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.758297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.758530 (+ 233us) server_negotiation.cc:407] Connection header received
0806 02:07:53.758617 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.758620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.758655 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.758724 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.759279 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58402: BlockingRecv error: recv got EOF from 127.0.0.1:58402 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:07:53.759517 28756 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.760533 28756 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.761406 28756 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:53.770035 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.762208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.28.21.1:43211)
0806 02:07:53.762330 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.762334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.762460 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:07:53.762613 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.762616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.762660 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.762745 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.763237 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.763747 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.764532 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.764766 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.764872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.767131 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.767135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.767137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.767156 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.768376 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.768784 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.768788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.768790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.768867 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.769180 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.769183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.769185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.769353 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.769543 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.769786 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.769899 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 02:07:53.772528 2744 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:07:53.771581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.28.21.1:43211)
0806 02:07:53.771699 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.771703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.771817 (+ 114us) server_negotiation.cc:407] Connection header received
0806 02:07:53.771927 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.771930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.771968 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.772040 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.772409 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58422: BlockingRecv error: recv got EOF from 127.0.0.1:58422 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:07:53.772687 28756 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785982043785074-28756-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.773687 28756 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:53.774613 28756 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:07:53.783136 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:53.775368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.28.21.1:43211)
0806 02:07:53.775484 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:53.775488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:53.775587 (+ 99us) server_negotiation.cc:407] Connection header received
0806 02:07:53.775727 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:53.775730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:53.775773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:53.775851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:53.776370 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.776837 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.777615 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:53.777832 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:53.777937 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:53.780224 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:53.780227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:53.780230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:53.780253 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:53.781484 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.781917 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.781922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.781923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.782004 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:53.782279 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:53.782283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:53.782285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:53.782518 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:53.782647 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:53.782883 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:07:53.782988 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:07:53.784147 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2312
I20260806 02:07:53.791383 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2454
I20260806 02:07:53.798818 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2598
I20260806 02:07:53.805572 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2216
2026-08-06T02:07:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2233 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2757](info): set up 2 sockets
Aug 06 02:07:53 dist-test-slave-gp6n krb5kdc[2757](info): commencing operation
krb5kdc: starting...
W20260806 02:07:54.893611 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:07:54 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:07:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab.
Entry for principal oryx/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab.
I20260806 02:07:55.025439 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41525
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab
--principal=oryx/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:55.139132 2771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:55.139448 2771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:55.139530 2771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:55.143500 2771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:55.143589 2771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:55.143626 2771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:55.143666 2771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:55.143699 2771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:55.149113 2771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab
--principal=oryx/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:55.150234 2771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:55.151371 2771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:07:55.160463 2771 server_base.cc:1034] running on GCE node
W20260806 02:07:55.160393 2779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.160373 2776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.160403 2777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.161178 2771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:55.162376 2771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:55.163619 2771 hybrid_clock.cc:648] HybridClock initialized: now 1785982075163599 us; error 39 us; skew 500 ppm
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:55.166352 2771 init.cc:377] Logged in from keytab as oryx/127.28.21.62@KRBTEST.COM (short username oryx)
I20260806 02:07:55.167307 2771 webserver.cc:460] Webserver started at http://127.28.21.62:40957/ using document root <none> and password file <none>
I20260806 02:07:55.167881 2771 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:55.167971 2771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:55.168210 2771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:55.170264 2771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "41c9b010f7f743c1a3d34c47562c7770"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "ee55b27637c85d8962c3187ad11da412"
server_key_iv: "af3498390837fe6393a832b6d85a208a"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.170912 2771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "41c9b010f7f743c1a3d34c47562c7770"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "ee55b27637c85d8962c3187ad11da412"
server_key_iv: "af3498390837fe6393a832b6d85a208a"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.174801 2771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:07:55.177356 2786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.178728 2771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:55.178881 2771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "41c9b010f7f743c1a3d34c47562c7770"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "ee55b27637c85d8962c3187ad11da412"
server_key_iv: "af3498390837fe6393a832b6d85a208a"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.179003 2771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:55.196869 2771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:55.200310 2771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:55.200462 2771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:55.207413 2771 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:41525
I20260806 02:07:55.207734 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:41525 every 8 connection(s)
I20260806 02:07:55.208400 2771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:07:55.211109 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:55.212595 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2771
I20260806 02:07:55.212700 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:07:55.213021 28756 external_mini_cluster.cc:1428] Setting key c47f985c1de277a348e93250fb378e38
I20260806 02:07:55.217758 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Bootstrap starting.
I20260806 02:07:55.220301 2843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:55.221199 2843 log.cc:824] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:55.223098 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: No bootstrap required, opened a new log
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:55.227283 2843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } }
I20260806 02:07:55.227563 2843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:55.227638 2843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c9b010f7f743c1a3d34c47562c7770, State: Initialized, Role: FOLLOWER
I20260806 02:07:55.228114 2843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } }
I20260806 02:07:55.228272 2843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:07:55.228345 2843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:07:55.228459 2843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:55.229573 2843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } }
I20260806 02:07:55.229928 2843 leader_election.cc:302] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c9b010f7f743c1a3d34c47562c7770; no voters:
I20260806 02:07:55.230238 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.214499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.28.21.62:41525)
0806 02:07:55.214911 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:55.214932 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:55.214989 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:07:55.215848 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:55.215872 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:55.216303 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:55.216667 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:55.217345 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.218812 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.219853 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.220587 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.220743 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.223777 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:55.223784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:55.223799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:55.223830 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:55.226769 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.227221 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.227227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.227233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.227380 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.227798 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.227801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.227803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.228305 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:55.228506 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:55.228859 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:07:55.229166 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260806 02:07:55.230258 2843 leader_election.cc:288] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:07:55.230381 2848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:55.230628 2848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 LEADER]: Becoming Leader. State: Replica: 41c9b010f7f743c1a3d34c47562c7770, State: Running, Role: LEADER
I20260806 02:07:55.230945 2848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } }
I20260806 02:07:55.231541 2843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:55.232146 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c9b010f7f743c1a3d34c47562c7770. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } }
I20260806 02:07:55.232256 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:55.232935 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } }
I20260806 02:07:55.233047 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:55.233342 2854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:55.236356 2854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:55.241833 2854 catalog_manager.cc:1347] Generated new cluster ID: 39cbdd7d61684832a5faf54edf9cfc2f
I20260806 02:07:55.241900 2854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:55.264441 2854 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:07:55.266819 2854 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:55.273124 2854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Generated new TSK 0
I20260806 02:07:55.274026 2854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
I20260806 02:07:55.318528 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:55.433133 2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:55.433431 2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:55.433511 2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:55.437673 2869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:55.437767 2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:55.437862 2869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:55.443216 2869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:55.444344 2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:55.445458 2869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:55.455639 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.455682 2869 server_base.cc:1034] running on GCE node
W20260806 02:07:55.455632 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.455632 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.456313 2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:55.456959 2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:55.458175 2869 hybrid_clock.cc:648] HybridClock initialized: now 1785982075458139 us; error 46 us; skew 500 ppm
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:55.461011 2869 init.cc:377] Logged in from keytab as oryx/127.28.21.1@KRBTEST.COM (short username oryx)
I20260806 02:07:55.461879 2869 webserver.cc:460] Webserver started at http://127.28.21.1:38281/ using document root <none> and password file <none>
I20260806 02:07:55.462488 2869 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:55.462559 2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:55.462734 2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:55.464969 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "6438ccb1f4724712b1785f7078d17c77"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "fa1f5da0b392dc380dab56ead22e093d"
server_key_iv: "3b27df776ae77d404f805c02113c5de5"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.465504 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "6438ccb1f4724712b1785f7078d17c77"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "fa1f5da0b392dc380dab56ead22e093d"
server_key_iv: "3b27df776ae77d404f805c02113c5de5"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.469259 2869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:07:55.471792 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.473048 2869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:55.473148 2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "6438ccb1f4724712b1785f7078d17c77"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "fa1f5da0b392dc380dab56ead22e093d"
server_key_iv: "3b27df776ae77d404f805c02113c5de5"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.473302 2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:55.490087 2869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:55.493032 2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:55.493237 2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:55.493839 2869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:55.494720 2869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:55.494767 2869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.494835 2869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:55.494865 2869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.503739 2869 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:40969
I20260806 02:07:55.503772 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:40969 every 8 connection(s)
I20260806 02:07:55.504616 2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:55.505637 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 2869
I20260806 02:07:55.505762 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:07:55.506062 28756 external_mini_cluster.cc:1428] Setting key d035778a99b8f61227817cc0f8042317
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.2@KRBTEST.COM" created.
I20260806 02:07:55.519577 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.506497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:40277 (local address 127.28.21.62:41525)
0806 02:07:55.506641 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:55.506645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:55.507508 (+ 863us) server_negotiation.cc:407] Connection header received
0806 02:07:55.508957 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:55.508960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:55.509015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:55.509095 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:55.510272 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.511159 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.512421 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.512744 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.512832 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.515665 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:55.515670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:55.515672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:55.515705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:55.517307 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.517957 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.517962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.517964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.518060 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.518421 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.518424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.518426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.518606 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:55.518750 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:55.519249 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 02:07:55.519403 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:55.520567 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.506897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:40277)
0806 02:07:55.507361 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:55.507414 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:55.508740 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:55.509268 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:55.509275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:55.509642 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:55.510126 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.510139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.511293 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.511296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.512292 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.512301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.512932 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.512939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.513176 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.513804 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:55.513829 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:55.515502 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:55.517429 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.517435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.517449 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.517852 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.518154 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.518157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.518159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.518308 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.518748 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:55.518755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:55.519016 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 02:07:55.519207 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 02:07:55.519530 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":144,"threads_started":1}
I20260806 02:07:55.521950 3004 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:55.522197 3004 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:55.522765 3004 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:55.524279 2806 ts_manager.cc:194] Registered new tserver with Master: 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:55.525395 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:40277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
I20260806 02:07:55.535439 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:55.644012 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:55.644296 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:55.644374 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:55.648312 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:55.648401 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:55.648489 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:55.653537 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:55.654574 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:55.655568 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:55.665257 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.665330 3009 server_base.cc:1034] running on GCE node
W20260806 02:07:55.665272 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.665303 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.665958 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:55.666644 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:55.667809 3009 hybrid_clock.cc:648] HybridClock initialized: now 1785982075667793 us; error 28 us; skew 500 ppm
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:55.670671 3009 init.cc:377] Logged in from keytab as oryx/127.28.21.2@KRBTEST.COM (short username oryx)
I20260806 02:07:55.671583 3009 webserver.cc:460] Webserver started at http://127.28.21.2:43633/ using document root <none> and password file <none>
I20260806 02:07:55.672147 3009 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:55.672245 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:55.672484 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:55.674552 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "a8b7fad6f52648348de44ad4e87a09ff"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "6621b4ce4d75064a48f136cd19406d38"
server_key_iv: "24144d8d8b8cf7c966fc2096b113b48c"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.675112 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "a8b7fad6f52648348de44ad4e87a09ff"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "6621b4ce4d75064a48f136cd19406d38"
server_key_iv: "24144d8d8b8cf7c966fc2096b113b48c"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.678993 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:07:55.681267 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.682749 3009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:55.682904 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a8b7fad6f52648348de44ad4e87a09ff"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "6621b4ce4d75064a48f136cd19406d38"
server_key_iv: "24144d8d8b8cf7c966fc2096b113b48c"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.683014 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:55.700596 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:55.703648 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:55.703851 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:55.704427 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:55.705240 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:55.705287 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.705351 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:55.705385 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.714457 3009 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:37723
I20260806 02:07:55.714512 3142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:37723 every 8 connection(s)
I20260806 02:07:55.715369 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:55.721891 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3009
I20260806 02:07:55.722034 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:07:55.722355 28756 external_mini_cluster.cc:1428] Setting key 4c0b9ee4675f2c6062db1ce7336a4712
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:55.730258 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.717209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:42215 (local address 127.28.21.62:41525)
0806 02:07:55.717408 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:55.717412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:55.718213 (+ 801us) server_negotiation.cc:407] Connection header received
0806 02:07:55.719671 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:55.719674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:55.719733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:55.719846 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:55.721013 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.721570 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.723171 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.723412 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.723529 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.726269 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:55.726272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:55.726275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:55.726301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:55.727832 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.728479 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.728481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.728483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.728561 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.728969 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.728971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.728972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.729232 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:55.729365 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:55.730036 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 02:07:55.730139 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:07:55.731490 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.717577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:42215)
0806 02:07:55.718049 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:55.718094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:55.719455 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:55.720029 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:55.720036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:55.720396 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:55.720863 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.720877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.721736 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.721739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.723041 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.723051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.723567 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.723571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.723742 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.724403 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:55.724419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:55.726072 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:55.727955 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.727964 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.727982 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.728375 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.728662 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.728665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.728668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.728872 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.729379 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:55.729389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:55.729736 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 02:07:55.729982 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:07:55.730365 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for oryx/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:07:55.732832 3143 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:55.733125 3143 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:55.733675 3143 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:55.734760 2806 ts_manager.cc:194] Registered new tserver with Master: a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:55.735339 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:42215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
I20260806 02:07:55.754128 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:55.864169 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:55.864413 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:55.864492 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:55.868417 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:55.868466 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:55.868557 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:55.874253 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:55.875309 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:55.876293 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:55.885845 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.885850 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:55.885905 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:55.886094 3148 server_base.cc:1034] running on GCE node
I20260806 02:07:55.886662 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:55.887319 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:55.888499 3148 hybrid_clock.cc:648] HybridClock initialized: now 1785982075888474 us; error 48 us; skew 500 ppm
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:55.891291 3148 init.cc:377] Logged in from keytab as oryx/127.28.21.3@KRBTEST.COM (short username oryx)
I20260806 02:07:55.892132 3148 webserver.cc:460] Webserver started at http://127.28.21.3:39253/ using document root <none> and password file <none>
I20260806 02:07:55.892684 3148 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:55.892772 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:55.893002 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:55.895076 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "a15b8017a63e48ea87bee7e2965b08d7"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "3576709b17704ff9093eee3c7714b99f"
server_key_iv: "cdedd5f064edf93fb8f4d7e13fb20119"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.895653 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "a15b8017a63e48ea87bee7e2965b08d7"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "3576709b17704ff9093eee3c7714b99f"
server_key_iv: "cdedd5f064edf93fb8f4d7e13fb20119"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.899298 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:55.901620 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.902850 3148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:55.902976 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a15b8017a63e48ea87bee7e2965b08d7"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "3576709b17704ff9093eee3c7714b99f"
server_key_iv: "cdedd5f064edf93fb8f4d7e13fb20119"
server_key_version: "encryptionkey@0"
I20260806 02:07:55.903085 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:55.915732 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:55.918926 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:55.919121 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:55.919657 3148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:55.920431 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:55.920508 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.920578 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:55.920629 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:55.929077 3148 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:41383
I20260806 02:07:55.929100 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:41383 every 8 connection(s)
I20260806 02:07:55.929947 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:55.930712 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3148
I20260806 02:07:55.930848 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:07:55.931136 28756 external_mini_cluster.cc:1428] Setting key 1f5c5ab13d5a65d32314c4165d3e93b5
Aug 06 02:07:55 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:55.944607 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.932812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:54077 (local address 127.28.21.62:41525)
0806 02:07:55.932975 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:55.932979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:55.934023 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 02:07:55.935435 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:55.935438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:55.935494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:55.935597 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:55.936766 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.937298 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.938509 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.938734 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.938846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.941179 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:55.941187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:55.941190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:55.941212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:55.942485 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.942990 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.942994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.942995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.943077 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:55.943416 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:55.943420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:55.943421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:55.943601 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:55.943738 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:55.944288 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 02:07:55.944442 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:07:55.945587 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:55.933365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:54077)
0806 02:07:55.933870 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:55.933919 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:55.935224 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:55.935771 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:55.935778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:55.936144 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:55.936618 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.936632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.937425 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.937429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.938322 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:55.938331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:55.938893 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:55.938898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:55.939067 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:55.939598 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:55.939613 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:55.941017 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:55.942596 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.942602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.942615 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.942873 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.943171 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:55.943174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:55.943176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:55.943313 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:55.943738 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:55.943746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:55.943990 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 02:07:55.944209 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 02:07:55.944552 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
I20260806 02:07:55.946699 3288 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:55.946950 3288 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:55.947422 3288 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:55.948439 2806 ts_manager.cc:194] Registered new tserver with Master: a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383)
I20260806 02:07:55.949003 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:54077
I20260806 02:07:55.956913 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.61.keytab.
Entry for principal oryx/127.28.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.61.keytab.
I20260806 02:07:55.985883 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:45155
--webserver_interface=127.28.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.61.keytab
--principal=oryx/127.28.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41525,127.28.21.61:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:07:56.097707 3293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:56.097955 3293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:56.097994 3293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:56.101920 3293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:56.101980 3293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:56.101999 3293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:56.102016 3293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:56.102031 3293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:56.107329 3293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41525,127.28.21.61:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.61.keytab
--principal=oryx/127.28.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
W20260806 02:07:56.107556 3293 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.21.62:41525,127.28.21.61:45155'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:07:56.108444 3293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:56.109503 3293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.118046 3298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.118060 3301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.118046 3299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.118420 3293 server_base.cc:1034] running on GCE node
I20260806 02:07:56.119128 3293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.120256 3293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.121430 3293 hybrid_clock.cc:648] HybridClock initialized: now 1785982076121410 us; error 28 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.124279 3293 init.cc:377] Logged in from keytab as oryx/127.28.21.61@KRBTEST.COM (short username oryx)
I20260806 02:07:56.125181 3293 webserver.cc:460] Webserver started at http://127.28.21.61:34419/ using document root <none> and password file <none>
I20260806 02:07:56.125761 3293 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.125854 3293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.126094 3293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:56.128225 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/instance:
uuid: "90286d46c56947e9a44bbb92505233c1"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "f6e453af6a5b05d23e8d2330fdbfc621"
server_key_iv: "f496436b9a944807c36ef894b6d9736e"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.128803 3293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal/instance:
uuid: "90286d46c56947e9a44bbb92505233c1"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "f6e453af6a5b05d23e8d2330fdbfc621"
server_key_iv: "f496436b9a944807c36ef894b6d9736e"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.132817 3293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:07:56.135433 3308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.136703 3293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:56.136847 3293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "90286d46c56947e9a44bbb92505233c1"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "f6e453af6a5b05d23e8d2330fdbfc621"
server_key_iv: "f496436b9a944807c36ef894b6d9736e"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.136961 3293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.156184 3293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:56.159765 3293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:56.159979 3293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:56.167309 3293 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.61:45155
I20260806 02:07:56.167593 3366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.61:45155 every 8 connection(s)
I20260806 02:07:56.168301 3293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
I20260806 02:07:56.171093 3367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:56.172566 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3293
I20260806 02:07:56.175316 3367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.61@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:56.187196 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.177383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.28.21.62:41525)
0806 02:07:56.177539 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.177544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.177789 (+ 245us) server_negotiation.cc:407] Connection header received
0806 02:07:56.177995 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.177999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.178050 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.178136 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.178824 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.179347 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.180273 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.180511 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.180624 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.183517 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.183521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.183524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.183551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.185310 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.185753 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.185757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.185759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.185844 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.186136 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.186140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.186142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.186351 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.186507 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.186870 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.186984 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:07:56.187978 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.174151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.28.21.61:45155)
0806 02:07:56.174609 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.174628 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.174670 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:07:56.175517 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.175543 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.175965 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.176321 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.177174 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.178630 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.179486 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.179966 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.180138 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.182941 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.182949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.182968 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.182989 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.185075 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.185495 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.185508 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.185509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.185616 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.185898 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.185903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.185904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.186287 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.186600 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.186755 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.187095 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":317,"thread_start_us":108,"threads_started":1}
I20260806 02:07:56.187975 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.177305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:44984)
0806 02:07:56.177666 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.177714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.177882 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.178186 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.178191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.178578 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.178696 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.178702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.179507 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.179510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.180152 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.180160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.180657 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.180661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.180795 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.181356 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.181374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.183349 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.185415 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.185425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.185434 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.185651 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.185937 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.185941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.185944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.186054 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.186508 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.186515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.186660 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.186815 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.187178 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":113,"thread_start_us":137,"threads_started":1}
I20260806 02:07:56.189400 3367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.61@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
I20260806 02:07:56.196877 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.189891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59028)
0806 02:07:56.190034 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.190049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.190118 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.190361 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.190363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.190554 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.190641 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.190647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.191708 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.191710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.192199 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.192206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.192611 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.192613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.192681 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.193118 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.193129 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.194239 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.195729 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.195732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.195733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.195912 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.196139 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.196141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.196142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.196213 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.196519 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.196521 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.196588 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.196666 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.196751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:07:56.196942 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.190040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.28.21.61:45155)
0806 02:07:56.190143 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.190148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.190181 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:07:56.190217 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.190220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.190260 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.190338 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.190736 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.191615 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.192293 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.192519 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.192595 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.194351 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.194355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.194357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.194377 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.195630 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.195993 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.195995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.195996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.196064 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.196285 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.196287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.196288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.196442 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.196515 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.196719 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.196802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:07:56.199590 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Bootstrap starting.
I20260806 02:07:56.201758 3367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:56.202520 3367 log.cc:824] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:56.204397 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: No bootstrap required, opened a new log
I20260806 02:07:56.207633 3367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } }
I20260806 02:07:56.207818 3367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:56.207873 3367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90286d46c56947e9a44bbb92505233c1, State: Initialized, Role: FOLLOWER
I20260806 02:07:56.208343 3367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } }
I20260806 02:07:56.208841 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } } }
I20260806 02:07:56.208987 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:56.209527 3367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:07:56.214457 3385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:56.214566 3385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:56.218583 3293 master_runner.cc:428] Detected that this master 90286d46c56947e9a44bbb92505233c1 is joining an existing cluster
I20260806 02:07:56.218664 3293 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.21.61:45155
I20260806 02:07:56.231936 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.222269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.28.21.62:41525)
0806 02:07:56.222427 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.222432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.223049 (+ 617us) server_negotiation.cc:407] Connection header received
0806 02:07:56.223263 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.223267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.223315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.223366 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.224052 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.224902 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.226195 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.226567 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.226647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.228067 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.228071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.228074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.228100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.229961 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.230535 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.230538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.230540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.230629 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.230928 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.230931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.230933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.231152 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.231224 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.231628 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.231749 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:07:56.232513 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.222534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:44986)
0806 02:07:56.222958 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.222974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.223147 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.223540 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.223542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.223768 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.223914 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.223922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.225065 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.225068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.226050 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.226064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.226814 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.226818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.226912 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.227431 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.227444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.227900 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.230084 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.230087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.230089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.230422 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.230740 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.230743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.230745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.230829 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.231294 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.231297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.231447 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.232182 (+ 735us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.232325 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":122,"threads_started":1}
I20260806 02:07:56.241976 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.223240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.28.21.61:45155)
0806 02:07:56.223393 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.223397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.227305 (+ 3908us) server_negotiation.cc:407] Connection header received
0806 02:07:56.228549 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.228553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.228611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.228664 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.233296 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.234036 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.235205 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.235544 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.235665 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.237005 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.237009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.237012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.237031 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.239798 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.240353 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.240360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.240361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.240458 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.241021 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.241024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.241026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.241264 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.241346 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.241684 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.241796 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:07:56.242547 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.222764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59032)
0806 02:07:56.227222 (+ 4458us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.227235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.228407 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.229139 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.229142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.233045 (+ 3903us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.233185 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.233193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.234165 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.234169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.235078 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.235088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.235699 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.235703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.235804 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.236468 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.236482 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.236879 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.239908 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.239912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.239914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.240243 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.240557 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.240560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.240562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.240925 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.241429 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.241432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.241521 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.242098 (+ 577us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.242245 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4398,"thread_start_us":88,"threads_started":1}
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.62@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
I20260806 02:07:56.250053 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.237763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.28.21.61:45155)
0806 02:07:56.238049 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.238058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.238903 (+ 845us) server_negotiation.cc:407] Connection header received
0806 02:07:56.238982 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.238985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.239046 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.239137 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.241047 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.241855 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.243163 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.243587 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.243706 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.246213 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.246217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.246220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.246244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.248165 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.248711 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.248715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.248717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.248843 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.249177 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.249180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.249181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.249367 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.249461 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.249828 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.249924 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260806 02:07:56.250151 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.237751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59042)
0806 02:07:56.238604 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.238657 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.238782 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.239970 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.239976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.240770 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.240911 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.240919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.241993 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.241996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.243033 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.243044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.243813 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.243817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.243936 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.244538 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.244558 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.246090 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.248289 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.248298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.248304 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.248597 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.248958 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.248962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.248964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.249062 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.249483 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.249490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.249627 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.249776 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.249974 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":105,"threads_started":1}
I20260806 02:07:56.251547 2806 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.21.61:45155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "90286d46c56947e9a44bbb92505233c1"
member_type: NON_VOTER
last_known_addr {
host: "127.28.21.61"
port: 45155
}
attrs {
promote: true
}
}
dest_uuid: "41c9b010f7f743c1a3d34c47562c7770"
cas_config_opid_index: -1
I20260806 02:07:56.252281 2806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } }
I20260806 02:07:56.253927 3397 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 90286d46c56947e9a44bbb92505233c1 (127.28.21.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.28.21.60@KRBTEST.COM; defaulting to no policy
I20260806 02:07:56.254855 3338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:56.256098 3397 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.21.61:45155
I20260806 02:07:56.256425 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } }
I20260806 02:07:56.256536 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:56.256723 3293 master.cc:537] Master@127.28.21.61:45155 shutting down...
I20260806 02:07:56.257547 3338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.60@KRBTEST.COM" created.
I20260806 02:07:56.258273 3399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:07:56.260589 2793 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 02:07:56.261569 2793 consensus_peers.cc:487] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 -> Peer 90286d46c56947e9a44bbb92505233c1 (127.28.21.61:45155): Couldn't send request to peer 90286d46c56947e9a44bbb92505233c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:07:56.261698 3293 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:56.261930 3293 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:56.262004 3293 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: stopping tablet replica
I20260806 02:07:56.277202 3293 master.cc:552] Master@127.28.21.61:45155 shutdown complete.
I20260806 02:07:56.277249 3293 master_runner.cc:305] Clearing existing system tablet
I20260806 02:07:56.277545 3293 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.277609 3293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.278949 3293 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 02:07:56.279876 3404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.280153 3293 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.280277 3293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:07:56.280350 3293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "90286d46c56947e9a44bbb92505233c1"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "f6e453af6a5b05d23e8d2330fdbfc621"
server_key_iv: "f496436b9a944807c36ef894b6d9736e"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.282955 3293 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:56.286000 3293 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:07:56.286074 3293 log.cc:1196] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.60.keytab.
Entry for principal oryx/127.28.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.60.keytab.
I20260806 02:07:56.286443 3293 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Deleting consensus metadata
I20260806 02:07:56.286643 3293 master_runner.cc:315] Copying system tablet from 127.28.21.62:41525
I20260806 02:07:56.287285 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:33711
--webserver_interface=127.28.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.60.keytab
--principal=oryx/127.28.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:07:56.290122 3293 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: tablet copy: Beginning tablet copy session from remote peer at address 127.28.21.62:41525
I20260806 02:07:56.305788 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.290790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:44996)
0806 02:07:56.298607 (+ 7817us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.298622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.298784 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.299092 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.299095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.299310 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.299423 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.299430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.300212 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.300217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.300842 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.300851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.301327 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.301331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.301428 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.301992 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.302012 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.302341 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.304187 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.304192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.304194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.304409 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.304705 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.304710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.304712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.304806 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.305201 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.305204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.305314 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.305430 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.305581 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7741,"thread_start_us":113,"threads_started":1}
I20260806 02:07:56.305780 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.290859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.28.21.62:41525)
0806 02:07:56.290993 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.290998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.298728 (+ 7730us) server_negotiation.cc:407] Connection header received
0806 02:07:56.298919 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.298922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.298979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.299074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.299555 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.300093 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.300977 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.301205 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.301319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.302530 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.302534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.302537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.302570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.304068 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.304512 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.304515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.304518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.304610 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.304906 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.304911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.304913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.305101 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.305213 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.305491 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.305610 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:07:56.307016 2826 tablet_copy_service.cc:140] P 41c9b010f7f743c1a3d34c47562c7770: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 90286d46c56947e9a44bbb92505233c1 ({username='oryx', principal='oryx/127.28.21.61@KRBTEST.COM'} at 127.0.0.1:44996)
I20260806 02:07:56.307212 2826 tablet_copy_service.cc:161] P 41c9b010f7f743c1a3d34c47562c7770: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 90286d46c56947e9a44bbb92505233c1 at {username='oryx', principal='oryx/127.28.21.61@KRBTEST.COM'} at 127.0.0.1:44996: session id = 90286d46c56947e9a44bbb92505233c1-00000000000000000000000000000000
I20260806 02:07:56.308856 2826 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:07:56.311046 3293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:07:56.313230 3293 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: tablet copy: Starting download of 0 data blocks...
I20260806 02:07:56.313444 3293 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: tablet copy: Starting download of 1 WAL segments...
I20260806 02:07:56.315559 3293 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:07:56.317076 2826 tablet_copy_service.cc:342] P 41c9b010f7f743c1a3d34c47562c7770: Request end of tablet copy session 90286d46c56947e9a44bbb92505233c1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.21.61@KRBTEST.COM'} at 127.0.0.1:44996
I20260806 02:07:56.317204 2826 tablet_copy_service.cc:433] P 41c9b010f7f743c1a3d34c47562c7770: ending tablet copy session 90286d46c56947e9a44bbb92505233c1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 90286d46c56947e9a44bbb92505233c1
I20260806 02:07:56.325217 3293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.327050 3412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.327068 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.327369 3413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.331450 3293 server_base.cc:1034] running on GCE node
I20260806 02:07:56.331641 3293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.332026 3293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.333179 3293 hybrid_clock.cc:648] HybridClock initialized: now 1785982076333165 us; error 28 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.334781 3293 init.cc:377] Logged in from keytab as oryx/127.28.21.61@KRBTEST.COM (short username oryx)
I20260806 02:07:56.335273 3293 webserver.cc:460] Webserver started at http://127.28.21.61:33039/ using document root <none> and password file <none>
I20260806 02:07:56.335489 3293 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.335543 3293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.336647 3293 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.337581 3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.337809 3293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.337904 3293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "90286d46c56947e9a44bbb92505233c1"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "f6e453af6a5b05d23e8d2330fdbfc621"
server_key_iv: "f496436b9a944807c36ef894b6d9736e"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.338003 3293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.343272 3293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:56.345168 3293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:56.350132 3293 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.61:45155
I20260806 02:07:56.350668 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.61:45155 every 8 connection(s)
I20260806 02:07:56.350931 3293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
I20260806 02:07:56.352483 3477 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:07:56.353364 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Bootstrap starting.
I20260806 02:07:56.361742 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:07:56.362092 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: Bootstrap complete.
I20260806 02:07:56.362697 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } }
I20260806 02:07:56.362854 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90286d46c56947e9a44bbb92505233c1, State: Initialized, Role: LEARNER
I20260806 02:07:56.362975 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } }
I20260806 02:07:56.363359 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } }
I20260806 02:07:56.363510 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:56.363526 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:07:56.366741 3492 catalog_manager.cc:1259] Loaded cluster ID: 39cbdd7d61684832a5faf54edf9cfc2f
I20260806 02:07:56.366868 3492 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: loading cluster ID for follower catalog manager: success
I20260806 02:07:56.368417 3492 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: acquiring CA information for follower catalog manager: success
I20260806 02:07:56.369596 3492 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:07:56.434108 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:56.434366 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:56.434465 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:56.438488 3405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:07:56.438555 3405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:56.438575 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:56.438591 3405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:07:56.438644 3405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:07:56.443859 3405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.60.keytab
--principal=oryx/127.28.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:33711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:56.444908 3405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:56.445966 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.454864 3498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.454860 3500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.454860 3497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.455186 3405 server_base.cc:1034] running on GCE node
I20260806 02:07:56.455783 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.456911 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.458112 3405 hybrid_clock.cc:648] HybridClock initialized: now 1785982076458080 us; error 41 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.461081 3405 init.cc:377] Logged in from keytab as oryx/127.28.21.60@KRBTEST.COM (short username oryx)
I20260806 02:07:56.462260 3405 webserver.cc:460] Webserver started at http://127.28.21.60:46485/ using document root <none> and password file <none>
I20260806 02:07:56.462859 3405 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.462926 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.463150 3405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:07:56.465359 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/instance:
uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "94698922764927730202452b301b8103"
server_key_iv: "5c80326d9d47ee87a7fc7942bc0c43e2"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.465975 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal/instance:
uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "94698922764927730202452b301b8103"
server_key_iv: "5c80326d9d47ee87a7fc7942bc0c43e2"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.469658 3405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:07:56.472146 3508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.473472 3405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:56.473615 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "94698922764927730202452b301b8103"
server_key_iv: "5c80326d9d47ee87a7fc7942bc0c43e2"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.473726 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.489436 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:56.492493 3405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:56.492695 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:56.499933 3405 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.60:33711
I20260806 02:07:56.500247 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.60:33711 every 8 connection(s)
I20260806 02:07:56.500926 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
I20260806 02:07:56.504141 3562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:56.505394 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3405
I20260806 02:07:56.510119 3562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.60@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:56.521138 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.507066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.28.21.60:33711)
0806 02:07:56.507543 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.507557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.507591 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:07:56.508284 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.508302 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.508617 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.508899 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.509913 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.511078 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.512242 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.512938 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.513112 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.515778 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.515785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.515797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.515827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.517862 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.518535 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.518546 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.518550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.518660 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.518983 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.518987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.518989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.519373 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.519582 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.519918 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.520185 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":143,"threads_started":1}
I20260806 02:07:56.523694 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.512222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.28.21.62:41525)
0806 02:07:56.512363 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.512368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.512671 (+ 303us) server_negotiation.cc:407] Connection header received
0806 02:07:56.512821 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.512825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.512882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.512980 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.513895 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.514540 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.515618 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.516050 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.516171 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.518823 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.518827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.518829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.518859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.520633 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.522183 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.522191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.522193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.522305 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.522672 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.522676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.522678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.522929 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.523042 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.523413 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.523534 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:07:56.523731 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.512002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45000)
0806 02:07:56.512528 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.512580 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.512710 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.513078 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.513084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.513561 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.513759 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.513768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.514690 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.514693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.515486 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.515495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.516214 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.516220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.516390 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.516995 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.517016 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.518663 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.520766 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.520773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.520779 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.521090 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.522461 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.522465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.522467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.522566 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.523068 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.523073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.523172 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.523351 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.523535 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":195,"threads_started":1}
I20260806 02:07:56.524848 3562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:07:56.527664 3004 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.60@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
I20260806 02:07:56.534101 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.525285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59044)
0806 02:07:56.525430 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.525446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.525516 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.525949 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.525952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.526105 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.526204 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.526210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.527189 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.527192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.527828 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.527843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.528493 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.528497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.528605 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.529132 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.529144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.530470 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.532405 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.532410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.532412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.532676 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.533022 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.533026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.533028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.533123 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.533541 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.533544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.533649 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.533764 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.533922 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:07:56.534123 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.525461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.28.21.61:45155)
0806 02:07:56.525706 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.525709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.525722 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:56.525758 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.525761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.525813 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.525894 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.526331 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.527062 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.527959 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.528327 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.528501 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.530630 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.530634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.530637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.530672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.532213 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.532801 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.532806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.532812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.532908 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.533221 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.533226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.533228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.533433 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.533551 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.533825 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.533957 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
I20260806 02:07:56.535024 3562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.60@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
I20260806 02:07:56.543303 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.535541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.0.0.1:56530)
0806 02:07:56.535654 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.535671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.535750 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.535944 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.535946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.536080 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.536148 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.536153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.537327 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.537331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.537920 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.537929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.538452 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.538458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.538532 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.538953 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.538972 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.540321 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.541878 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.541880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.541882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.542067 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.542332 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.542334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.542335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.542483 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.542901 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.542904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.542982 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.543077 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.543187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:07:56.543500 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.535637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.28.21.60:33711)
0806 02:07:56.535744 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.535748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.535766 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:07:56.535814 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.535816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.535847 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.535921 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.536254 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.537145 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.538039 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.538262 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.538388 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.540445 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.540448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.540450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.540470 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.541770 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.542163 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.542166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.542167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.542247 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.542571 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.542574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.542575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.542769 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.542859 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.543169 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.543270 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:07:56.546291 3562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Bootstrap starting.
I20260806 02:07:56.548599 3562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:56.549342 3562 log.cc:824] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:56.551294 3562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: No bootstrap required, opened a new log
I20260806 02:07:56.554128 3562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } }
I20260806 02:07:56.554365 3562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:56.554468 3562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1cc0a8dd354cc58016cb5844b44edb, State: Initialized, Role: FOLLOWER
I20260806 02:07:56.554885 3562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } }
I20260806 02:07:56.555342 3568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } } }
I20260806 02:07:56.555464 3568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:56.555997 3562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:07:56.561000 3580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:07:56.561091 3580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:07:56.564407 3405 master_runner.cc:428] Detected that this master 2d1cc0a8dd354cc58016cb5844b44edb is joining an existing cluster
I20260806 02:07:56.564512 3405 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.21.60:33711
I20260806 02:07:56.574776 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2869
I20260806 02:07:56.575443 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.566735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45014)
0806 02:07:56.567020 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.567036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.567147 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.567658 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.567661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.567898 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.568030 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.568041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.569118 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.569121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.570062 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.570070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.570768 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.570921 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.571422 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.571432 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.571745 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.573720 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.573723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.573725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.574042 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.574365 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.574368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.574370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.574484 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.574945 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.574948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.575037 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.575134 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.575261 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":35,"thread_start_us":118,"threads_started":1}
I20260806 02:07:56.575964 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.566716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.28.21.62:41525)
0806 02:07:56.566842 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.566846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.567130 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:07:56.567275 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.567282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.567340 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.567397 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.568169 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.568998 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570187 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.570563 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570644 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.571889 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.571894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.571896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.571922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.573590 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.574155 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.574160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.574162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.574253 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.574588 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.574592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.574594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.574800 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.574901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.575690 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.575803 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:56.585552 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:40969
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:41525
--webserver_port=38281
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.62@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
I20260806 02:07:56.597093 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.566754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.28.21.61:45155)
0806 02:07:56.566861 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.566866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.567353 (+ 487us) server_negotiation.cc:407] Connection header received
0806 02:07:56.567522 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.567526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.567581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.567633 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.568305 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.569107 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570317 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.570718 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570795 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.592115 (+ 21320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.592123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.592127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.592163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.594974 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.595509 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.595517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.595522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.595628 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.595975 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.595978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.595980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.596198 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.596311 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.596752 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.596904 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:07:56.597167 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.566957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59046)
0806 02:07:56.567267 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.567283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.567413 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.567782 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.567785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.568042 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.568164 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.568171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.569251 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.569255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.570181 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.570191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.570960 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.571190 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.591408 (+ 20218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.591425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.591932 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.595103 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.595106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.595108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.595383 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.595755 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.595758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.595760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.595857 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.596361 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.596364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.596460 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.596569 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.596972 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":29,"thread_start_us":108,"threads_started":1}
I20260806 02:07:56.597950 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.567355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.28.21.60:33711)
0806 02:07:56.567561 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.567565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.568090 (+ 525us) server_negotiation.cc:407] Connection header received
0806 02:07:56.568223 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.568226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.568258 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.568298 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.568782 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.569498 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570541 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.570886 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.570964 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.591945 (+ 20981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.591949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.591952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.591982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.593950 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.594522 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.594526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.594528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.594617 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.594950 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.594953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.594955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.595192 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.595284 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.596755 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.597800 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":919,"server-negotiator.queue_time_us":130}
I20260806 02:07:56.598837 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.567464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.0.0.1:56544)
0806 02:07:56.568014 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.568027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.568140 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.568394 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.568397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.568586 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.568678 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.568684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.569628 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.569630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.570438 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.570446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.571026 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.571029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.571145 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.591047 (+ 19902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.591069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.591759 (+ 690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.594081 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.594086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.594090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.594374 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.594734 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.594737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.594741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.594844 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.595337 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.595343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.595439 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.596826 (+ 1387us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.598672 (+ 1846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":1698,"thread_start_us":84,"threads_started":1}
I20260806 02:07:56.601958 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.583065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.28.21.60:33711)
0806 02:07:56.583348 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.583358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.583374 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:07:56.583445 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.583447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.583490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.583602 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.584084 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.584663 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.586396 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.586948 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.587572 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.589411 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.589414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.589417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.589439 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.599730 (+ 10291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.600279 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.600282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.600283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.600805 (+ 522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.601119 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.601124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.601126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.601307 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.601421 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.601684 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.601792 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
I20260806 02:07:56.601959 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.582743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.0.0.1:56556)
0806 02:07:56.582877 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.582892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.582968 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.583620 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.583624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.583824 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.583930 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.583940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.584920 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.584923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.585850 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.585865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.587106 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.587114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.587270 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.587855 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.587876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.589234 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.599918 (+ 10684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.599921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.599924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.600173 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.600926 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.600929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.600931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.601020 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.601411 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.601415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.601523 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.601633 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.601792 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 02:07:56.602977 2806 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.21.60:33711: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
member_type: NON_VOTER
last_known_addr {
host: "127.28.21.60"
port: 33711
}
attrs {
promote: true
}
}
dest_uuid: "41c9b010f7f743c1a3d34c47562c7770"
cas_config_opid_index: 5
I20260806 02:07:56.603555 2806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } }
I20260806 02:07:56.604638 3398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.605904 3398 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.21.60:33711
I20260806 02:07:56.606081 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.606194 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:56.607088 3535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:56.607204 3405 master.cc:537] Master@127.28.21.60:33711 shutting down...
I20260806 02:07:56.608690 3535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:07:56.610499 3397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:07:56.612511 2793 consensus_peers.cc:487] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 -> Peer 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60:33711): Couldn't send request to peer 2d1cc0a8dd354cc58016cb5844b44edb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:07:56.612705 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.606634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59050)
0806 02:07:56.607990 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.608005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.608087 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.608546 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.608549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.608566 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:56.608652 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.608658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.609715 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.609721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.611011 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.611021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.612154 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.612157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.612268 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.612332 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.612428 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.612518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283}
I20260806 02:07:56.613085 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.607118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.28.21.61:45155)
0806 02:07:56.607892 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.607896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.608173 (+ 277us) server_negotiation.cc:407] Connection header received
0806 02:07:56.608226 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.608229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.608286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.608379 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:56.608786 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.609581 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.611152 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.611816 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.611922 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.612090 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.612866 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.612951 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686}
I20260806 02:07:56.613966 3450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 LEARNER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:07:56.614495 3397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:07:56.616328 3450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.618299 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c9b010f7f743c1a3d34c47562c7770. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.618484 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:07:56.618559 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: NON_VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: true } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.618604 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:07:56.619552 3397 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: attempting to promote NON_VOTER 90286d46c56947e9a44bbb92505233c1 to VOTER
I20260806 02:07:56.620085 3405 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:56.620049 3397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } }
I20260806 02:07:56.620348 3405 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:56.620617 3405 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: stopping tablet replica
I20260806 02:07:56.621639 3450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 LEARNER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260806 02:07:56.621917 2793 consensus_peers.cc:487] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 -> Peer 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60:33711): Couldn't send request to peer 2d1cc0a8dd354cc58016cb5844b44edb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:07:56.623001 3399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:07:56.625599 3399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90286d46c56947e9a44bbb92505233c1 (127.28.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.626968 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.627117 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:56.627260 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.627345 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:56.627931 3450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90286d46c56947e9a44bbb92505233c1 (127.28.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.629416 3592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.629578 3592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:56.636716 3405 master.cc:552] Master@127.28.21.60:33711 shutdown complete.
I20260806 02:07:56.636785 3405 master_runner.cc:305] Clearing existing system tablet
I20260806 02:07:56.637027 3405 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.637097 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.638334 3405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.639369 3594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.639689 3405 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.639876 3405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.639930 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "94698922764927730202452b301b8103"
server_key_iv: "5c80326d9d47ee87a7fc7942bc0c43e2"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.641675 3405 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:56.643808 3405 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:07:56.643919 3405 log.cc:1196] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:07:56.644178 3405 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Deleting consensus metadata
I20260806 02:07:56.644398 3405 master_runner.cc:315] Copying system tablet from 127.28.21.62:41525
I20260806 02:07:56.645582 3405 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: tablet copy: Beginning tablet copy session from remote peer at address 127.28.21.62:41525
I20260806 02:07:56.654346 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.646245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.28.21.62:41525)
0806 02:07:56.646443 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.646449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.646607 (+ 158us) server_negotiation.cc:407] Connection header received
0806 02:07:56.646783 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.646787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.646843 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.646937 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.647411 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.647997 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.648922 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.649226 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.649362 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.650837 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.650842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.650845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.650873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.652517 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.652998 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.653002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.653004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.653093 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.653405 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.653410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.653412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.653609 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.653727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.654045 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.654167 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:07:56.654376 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.646161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45028)
0806 02:07:56.646513 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.646526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.646664 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.646958 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.646961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.647184 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.647281 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.647289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.648115 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.648121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.648790 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.648799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.649364 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.649368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.649472 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.650109 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.650123 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.650652 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.652653 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.652656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.652659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.652885 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.653193 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.653198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.653200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.653303 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.653717 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.653720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.653840 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.653981 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.654156 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":94,"threads_started":1}
I20260806 02:07:56.655243 2826 tablet_copy_service.cc:140] P 41c9b010f7f743c1a3d34c47562c7770: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2d1cc0a8dd354cc58016cb5844b44edb ({username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:45028)
I20260806 02:07:56.655359 2826 tablet_copy_service.cc:161] P 41c9b010f7f743c1a3d34c47562c7770: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2d1cc0a8dd354cc58016cb5844b44edb at {username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:45028: session id = 2d1cc0a8dd354cc58016cb5844b44edb-00000000000000000000000000000000
I20260806 02:07:56.656428 2826 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:07:56.657969 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:07:56.660043 3405 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: tablet copy: Starting download of 0 data blocks...
I20260806 02:07:56.660261 3405 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: tablet copy: Starting download of 1 WAL segments...
I20260806 02:07:56.661753 3405 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:07:56.663570 2826 tablet_copy_service.cc:342] P 41c9b010f7f743c1a3d34c47562c7770: Request end of tablet copy session 2d1cc0a8dd354cc58016cb5844b44edb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:45028
I20260806 02:07:56.663690 2826 tablet_copy_service.cc:433] P 41c9b010f7f743c1a3d34c47562c7770: ending tablet copy session 2d1cc0a8dd354cc58016cb5844b44edb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2d1cc0a8dd354cc58016cb5844b44edb
I20260806 02:07:56.671093 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.672858 3602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.672928 3606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.673406 3603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.674602 3405 server_base.cc:1034] running on GCE node
I20260806 02:07:56.674798 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.675132 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.676286 3405 hybrid_clock.cc:648] HybridClock initialized: now 1785982076676292 us; error 46 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.677918 3405 init.cc:377] Logged in from keytab as oryx/127.28.21.60@KRBTEST.COM (short username oryx)
I20260806 02:07:56.678481 3405 webserver.cc:460] Webserver started at http://127.28.21.60:42043/ using document root <none> and password file <none>
I20260806 02:07:56.678702 3405 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.678766 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.679986 3405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.681000 3613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.681411 3405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:56.681515 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "2d1cc0a8dd354cc58016cb5844b44edb"
format_stamp: "Formatted at 2026-08-06 02:07:56 on dist-test-slave-gp6n"
server_key: "94698922764927730202452b301b8103"
server_key_iv: "5c80326d9d47ee87a7fc7942bc0c43e2"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.681629 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.703934 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:56.706259 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:56.712119 3405 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.60:33711
I20260806 02:07:56.712136 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.60:33711 every 8 connection(s)
I20260806 02:07:56.712756 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
I20260806 02:07:56.714564 3668 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:07:56.715801 3668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Bootstrap starting.
I20260806 02:07:56.726517 3668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:07:56.726846 3668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Bootstrap complete.
I20260806 02:07:56.727397 3668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } }
I20260806 02:07:56.727548 3668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d1cc0a8dd354cc58016cb5844b44edb, State: Initialized, Role: LEARNER
I20260806 02:07:56.727677 3668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } }
I20260806 02:07:56.728073 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:56.728236 3668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:07:56.728272 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: LEARNER
I20260806 02:07:56.731657 3682 catalog_manager.cc:1259] Loaded cluster ID: 39cbdd7d61684832a5faf54edf9cfc2f
I20260806 02:07:56.731985 3682 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: loading cluster ID for follower catalog manager: success
I20260806 02:07:56.734163 3682 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: acquiring CA information for follower catalog manager: success
I20260806 02:07:56.735504 3682 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:07:56.738042 3143 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
W20260806 02:07:56.742051 3590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:56.742350 3590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:56.742429 3590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:56.748157 3590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:56.748220 3590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:56.748306 3590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:07:56.755992 3590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:40969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=38281
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:56.757234 3590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:56.758603 3590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.769784 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.769977 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.770234 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.770294 3590 server_base.cc:1034] running on GCE node
I20260806 02:07:56.770738 3590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.771427 3590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.772624 3590 hybrid_clock.cc:648] HybridClock initialized: now 1785982076772601 us; error 29 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.775408 3590 init.cc:377] Logged in from keytab as oryx/127.28.21.1@KRBTEST.COM (short username oryx)
I20260806 02:07:56.776341 3590 webserver.cc:460] Webserver started at http://127.28.21.1:38281/ using document root <none> and password file <none>
I20260806 02:07:56.776906 3590 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.777015 3590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.780844 3590 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:07:56.783001 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.784381 3590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:07:56.784521 3590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "6438ccb1f4724712b1785f7078d17c77"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "fa1f5da0b392dc380dab56ead22e093d"
server_key_iv: "3b27df776ae77d404f805c02113c5de5"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.785068 3590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:56.795935 3590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:56.798835 3590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:56.799029 3590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:56.799597 3590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:56.800419 3590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:56.800491 3590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.800561 3590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:56.800606 3590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.810330 3590 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:40969
I20260806 02:07:56.810376 3817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:40969 every 8 connection(s)
I20260806 02:07:56.811357 3590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:07:56.811822 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3590
I20260806 02:07:56.812040 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3009
I20260806 02:07:56.820657 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:37723
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:41525
--webserver_port=43633
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
I20260806 02:07:56.831331 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.815485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:38293 (local address 127.28.21.61:45155)
0806 02:07:56.815605 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.815609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.817249 (+ 1640us) server_negotiation.cc:407] Connection header received
0806 02:07:56.817466 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.817473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.817545 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.817611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.818773 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819735 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.821533 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.821962 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.822065 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.826885 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.826889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.826892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.826925 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.828773 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829365 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.829368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.829370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.829474 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829900 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.829903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.829905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.830125 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.830202 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.831024 (+ 822us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.831154 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:07:56.832440 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.814901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.28.21.1:39349)
0806 02:07:56.816779 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.816797 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.816914 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.817308 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.817311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.817653 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.818591 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.818611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819989 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.819993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.821360 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.821370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.822120 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.822124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.822490 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.823386 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.823407 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.826383 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.828723 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.828732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.828749 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.829230 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.829583 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.829588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.829783 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.830273 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.830279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.830705 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.830944 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.831283 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":488,"spinlock_wait_cycles":21376,"thread_start_us":113,"threads_started":1}
I20260806 02:07:56.832718 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.816521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:34949 (local address 127.28.21.62:41525)
0806 02:07:56.816676 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.816679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.816691 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:07:56.816732 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.816734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.816773 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.816877 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.818766 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819638 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.821555 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.821878 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.821952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.826307 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.826315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.826321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.826360 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.829795 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.830658 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.830664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.830668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.830778 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.831265 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.831273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.831278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.831620 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.831729 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.832251 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.832501 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:07:56.833276 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.815366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.28.21.1:38293)
0806 02:07:56.817155 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.817171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.817339 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.817800 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.817806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.818052 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.818587 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.818612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819896 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.819902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.821371 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.821387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.822262 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.822268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.822528 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.823461 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.823483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.826715 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.828888 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.828891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.828893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.829230 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.829588 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.829593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.829783 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.830273 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.830279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.830705 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.831859 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.832027 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":307,"spinlock_wait_cycles":28288,"thread_start_us":93,"threads_started":1}
I20260806 02:07:56.833395 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.813416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:39349 (local address 127.28.21.60:33711)
0806 02:07:56.815127 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:56.815134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:56.816983 (+ 1849us) server_negotiation.cc:407] Connection header received
0806 02:07:56.817033 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:56.817036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:56.817086 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:56.817172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:56.818767 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819869 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.821501 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.821864 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.821959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.826586 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:56.826590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:56.826593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:56.826624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:56.828575 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829363 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.829367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.829369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.829474 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:56.829905 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:56.829908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:56.829910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:56.830125 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:56.830215 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:56.833046 (+ 2831us) server_negotiation.cc:299] Negotiation successful
0806 02:07:56.833202 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1610,"thread_start_us":72,"threads_started":1}
I20260806 02:07:56.833509 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:56.813723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:34949)
0806 02:07:56.814300 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:56.814363 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:56.815911 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:56.817064 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:56.817073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:56.817659 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:56.818590 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.818611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.819809 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.819812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.821432 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:56.821443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:56.822120 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:56.822123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:56.822457 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:56.823386 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:56.823450 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:56.826020 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:56.830143 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.830150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.830156 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.830508 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.830926 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:56.830932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:56.830937 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:56.831053 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:56.831831 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:56.831834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:56.831998 (+ 164us) client_negotiation.cc:769] Sending connection context
0806 02:07:56.832995 (+ 997us) client_negotiation.cc:240] Negotiation successful
0806 02:07:56.833188 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":261,"spinlock_wait_cycles":22784,"thread_start_us":167,"threads_started":1}
I20260806 02:07:56.834450 3818 heartbeater.cc:347] Connected to a master server at 127.28.21.60:33711
I20260806 02:07:56.834487 3819 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:56.834535 3820 heartbeater.cc:347] Connected to a master server at 127.28.21.61:45155
I20260806 02:07:56.834699 3819 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:56.834698 3818 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:56.834698 3820 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:56.835343 3819 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:56.835341 3818 heartbeater.cc:511] Master 127.28.21.60:33711 requested a full tablet report, sending...
I20260806 02:07:56.835341 3820 heartbeater.cc:511] Master 127.28.21.61:45155 requested a full tablet report, sending...
I20260806 02:07:56.836726 2806 ts_manager.cc:194] Re-registered known tserver with Master: 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:56.836988 3630 ts_manager.cc:194] Registered new tserver with Master: 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:56.837483 3436 ts_manager.cc:194] Registered new tserver with Master: 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:56.837617 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:34949
I20260806 02:07:56.950970 3288 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
W20260806 02:07:56.955642 3826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:56.955884 3826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:56.955961 3826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:56.960311 3826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:56.960366 3826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:56.960454 3826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:07:56.966127 3826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:37723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=43633
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:56.967259 3826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:56.968273 3826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:56.978022 3834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.978022 3831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:56.977957 3832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:56.978541 3826 server_base.cc:1034] running on GCE node
I20260806 02:07:56.978931 3826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:56.979499 3826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:56.980706 3826 hybrid_clock.cc:648] HybridClock initialized: now 1785982076980670 us; error 60 us; skew 500 ppm
Aug 06 02:07:56 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:56.983620 3826 init.cc:377] Logged in from keytab as oryx/127.28.21.2@KRBTEST.COM (short username oryx)
I20260806 02:07:56.984445 3826 webserver.cc:460] Webserver started at http://127.28.21.2:43633/ using document root <none> and password file <none>
I20260806 02:07:56.984973 3826 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:56.985054 3826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:56.988864 3826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:07:56.991074 3841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:56.992336 3826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:56.992487 3826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a8b7fad6f52648348de44ad4e87a09ff"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "6621b4ce4d75064a48f136cd19406d38"
server_key_iv: "24144d8d8b8cf7c966fc2096b113b48c"
server_key_version: "encryptionkey@0"
I20260806 02:07:56.993038 3826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:57.021062 3826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:57.024286 3826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:57.024484 3826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:57.025106 3826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:57.025947 3826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:57.025993 3826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:57.026060 3826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:57.026091 3826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:57.036087 3826 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:37723
I20260806 02:07:57.036111 3960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:37723 every 8 connection(s)
I20260806 02:07:57.037083 3826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:07:57.039641 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3826
I20260806 02:07:57.039829 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3148
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982076, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
I20260806 02:07:57.054812 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:41383
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:41525
--webserver_port=39253
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:07:57.056459 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.039051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:38355 (local address 127.28.21.60:33711)
0806 02:07:57.039197 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.039202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.041340 (+ 2138us) server_negotiation.cc:407] Connection header received
0806 02:07:57.042561 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.042565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.042634 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.042682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.044228 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.044813 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.046784 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.047186 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.047276 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.051607 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.051615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.051621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.051657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.053823 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.054516 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.054520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.054522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.054616 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.055050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.055056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.055060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.055297 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.055384 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.056065 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.056219 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:07:57.056567 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.039043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:44381 (local address 127.28.21.62:41525)
0806 02:07:57.039184 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.039189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.041342 (+ 2153us) server_negotiation.cc:407] Connection header received
0806 02:07:57.042574 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.042581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.042659 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.042733 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.044262 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.045271 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.046899 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.047384 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.047491 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.051909 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.051917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.051924 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.052237 (+ 313us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.054330 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.054935 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.054941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.054945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.055051 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.055395 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.055399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.055401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.055650 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.055756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.056198 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.056335 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:07:57.058045 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.039409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.28.21.2:38355)
0806 02:07:57.041175 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.041234 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.042276 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.042968 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.042974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.043487 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.044072 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.044088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.044983 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.044990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.046642 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.046656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.047487 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.047494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.047757 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.048755 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.048783 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.051313 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.053965 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.053974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.053990 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.054354 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.054727 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.054732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.054737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.054929 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.055458 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.055467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.055772 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.056625 (+ 853us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.056982 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":110,"spinlock_wait_cycles":53376,"thread_start_us":1102,"threads_started":1}
I20260806 02:07:57.058367 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.039409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:44381)
0806 02:07:57.041179 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.041234 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.042293 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.042972 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.042978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.043519 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.044078 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.044094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.045408 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.045414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.046723 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.046738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.047690 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.047697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.047919 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.048755 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.048788 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.051713 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.054494 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.054497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.054500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.054788 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.055168 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.055173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.055179 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.055275 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.055838 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.055845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.055966 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.056864 (+ 898us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.057063 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"mutex_wait_us":147,"spinlock_wait_cycles":71168,"thread_start_us":134,"threads_started":1}
I20260806 02:07:57.059366 3961 heartbeater.cc:347] Connected to a master server at 127.28.21.60:33711
I20260806 02:07:57.059597 3962 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:57.059620 3961 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.059708 3962 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.060269 3961 heartbeater.cc:511] Master 127.28.21.60:33711 requested a full tablet report, sending...
I20260806 02:07:57.060269 3962 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:57.060726 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.048249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:55181 (local address 127.28.21.61:45155)
0806 02:07:57.048391 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.048397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.048414 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:07:57.048461 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.048466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.048523 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.048605 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.049306 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.050199 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.051518 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.051962 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.052065 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.055415 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.055419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.055422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.055453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.057928 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.059012 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.059016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.059018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.059129 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.059570 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.059577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.059581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.059849 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.059962 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.060342 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.060483 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:07:57.061326 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.040905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.28.21.2:55181)
0806 02:07:57.041399 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.041412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.042276 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.048740 (+ 6464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.048746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.049007 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.049131 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.049141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.050382 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.050387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.051383 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.051394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.052091 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.052097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.052224 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.052890 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.052907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.055230 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.058078 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.058083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.058087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.058875 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.059271 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.059275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.059277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.059397 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.060013 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.060019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.060146 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.060875 (+ 729us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.061049 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"spinlock_wait_cycles":52096,"thread_start_us":207,"threads_started":1}
I20260806 02:07:57.061506 2806 ts_manager.cc:194] Re-registered known tserver with Master: a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:57.061614 3630 ts_manager.cc:194] Registered new tserver with Master: a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:57.062247 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:44381
I20260806 02:07:57.062662 3963 heartbeater.cc:347] Connected to a master server at 127.28.21.61:45155
I20260806 02:07:57.062757 3963 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.063005 3963 heartbeater.cc:511] Master 127.28.21.61:45155 requested a full tablet report, sending...
I20260806 02:07:57.063807 3436 ts_manager.cc:194] Registered new tserver with Master: a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:57.156565 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.152942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.28.21.60:33711)
0806 02:07:57.153096 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.153101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.153229 (+ 128us) server_negotiation.cc:407] Connection header received
0806 02:07:57.153406 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.153409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.153489 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.153589 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.153871 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.154473 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.155532 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.155986 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.156108 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.156269 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.156343 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.156413 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:07:57.156569 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.152851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.0.0.1:56566)
0806 02:07:57.153124 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.153146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.153293 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.153618 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.153621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.153641 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.153757 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.153763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.154594 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.154600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.155394 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.155403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.156116 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.156120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.156220 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.156245 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.156320 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.156413 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260806 02:07:57.157358 3642 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:07:57.157740 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c9b010f7f743c1a3d34c47562c7770. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:57.157893 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: LEARNER
W20260806 02:07:57.171321 3968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:07:57.171571 3968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:07:57.171615 3968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:07:57.175724 3968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:07:57.175773 3968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:07:57.175830 3968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:07:57.180915 3968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:41383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=39253
--tserver_master_addrs=127.28.21.62:41525,127.28.21.61:45155,127.28.21.60:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:07:57.181943 3968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:07:57.183071 3968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:07:57.192840 3977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:57.192860 3975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:07:57.192930 3974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:07:57.193118 3968 server_base.cc:1034] running on GCE node
I20260806 02:07:57.193468 3968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:07:57.194051 3968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:07:57.195214 3968 hybrid_clock.cc:648] HybridClock initialized: now 1785982077195198 us; error 26 us; skew 500 ppm
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982077, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:07:57.197989 3968 init.cc:377] Logged in from keytab as oryx/127.28.21.3@KRBTEST.COM (short username oryx)
I20260806 02:07:57.198824 3968 webserver.cc:460] Webserver started at http://127.28.21.3:39253/ using document root <none> and password file <none>
I20260806 02:07:57.199321 3968 fs_manager.cc:359] Metadata directory not provided
I20260806 02:07:57.199390 3968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:07:57.203166 3968 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:07:57.205221 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:57.206521 3968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:07:57.206634 3968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a15b8017a63e48ea87bee7e2965b08d7"
format_stamp: "Formatted at 2026-08-06 02:07:55 on dist-test-slave-gp6n"
server_key: "3576709b17704ff9093eee3c7714b99f"
server_key_iv: "cdedd5f064edf93fb8f4d7e13fb20119"
server_key_version: "encryptionkey@0"
I20260806 02:07:57.207161 3968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:07:57.223696 3968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:07:57.227624 3968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:07:57.227922 3968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:07:57.228709 3968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:07:57.230007 3968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:07:57.230088 3968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:57.230150 3968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:07:57.230188 3968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:07:57.243700 3968 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:41383
I20260806 02:07:57.243758 4104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:41383 every 8 connection(s)
I20260806 02:07:57.244915 3968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:07:57.254354 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 3968
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982077, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.61@KRBTEST.COM
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982077, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.60@KRBTEST.COM
Aug 06 02:07:57 dist-test-slave-gp6n krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982077, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:07:57.262205 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.246840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:41961 (local address 127.28.21.62:41525)
0806 02:07:57.246987 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.246992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.248002 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 02:07:57.249221 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.249225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.249311 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.249394 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.250856 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251630 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253036 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.253373 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253445 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.257638 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.257641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.257644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.257675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.259773 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.260425 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.260429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.260430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.260523 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.260917 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.260923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.260926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.261130 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.261196 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.261913 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.262041 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:07:57.263051 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.247330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:53829 (local address 127.28.21.61:45155)
0806 02:07:57.247467 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.247474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.248074 (+ 600us) server_negotiation.cc:407] Connection header received
0806 02:07:57.249311 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.249316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.249390 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.249461 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.250840 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251376 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.252633 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.252869 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.252978 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.256905 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.256909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.256913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.256940 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.260460 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.260978 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.260982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.260984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.261074 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.261369 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.261372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.261374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.261571 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.261666 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.262758 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.262875 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:07:57.263485 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.247218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.28.21.3:53829)
0806 02:07:57.247990 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.248003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.248982 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.249571 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.249574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.250029 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.250685 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.250694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251527 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.251531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.252503 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.252512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.253010 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.253188 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.253819 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.253844 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.256701 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.260590 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.260594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.260596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.260846 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.261175 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.261178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.261180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.261264 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.261713 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.261716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.261837 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.261969 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.262333 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":183,"spinlock_wait_cycles":48640,"thread_start_us":149,"threads_started":1}
I20260806 02:07:57.263841 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.247217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:41961)
0806 02:07:57.247820 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.247895 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.248982 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.249559 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.249562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.250028 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.250688 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.250694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251768 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.251772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.252896 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.252906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.253599 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.253744 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.254327 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.254345 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.257507 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.259922 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.259928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.259943 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.260287 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.260632 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.260634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.260636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.260802 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.261278 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.261285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.261610 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.262373 (+ 763us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.262541 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":220,"spinlock_wait_cycles":13312,"thread_start_us":149,"threads_started":1}
I20260806 02:07:57.264848 4106 heartbeater.cc:347] Connected to a master server at 127.28.21.62:41525
I20260806 02:07:57.264961 4107 heartbeater.cc:347] Connected to a master server at 127.28.21.61:45155
I20260806 02:07:57.265105 4106 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.265726 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.248422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:42065 (local address 127.28.21.60:33711)
0806 02:07:57.248550 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.248555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.248626 (+ 71us) server_negotiation.cc:407] Connection header received
0806 02:07:57.249193 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.249196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.249252 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.249347 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.250810 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251630 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253035 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.253373 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253453 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.257677 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.257681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.257683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.257707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.259682 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.264344 (+ 4662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.264350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.264354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.264465 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.264800 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.264803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.264805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.265042 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.265129 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.265447 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.265578 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:07:57.265734 4106 heartbeater.cc:511] Master 127.28.21.62:41525 requested a full tablet report, sending...
I20260806 02:07:57.266179 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.248316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.28.21.3:42065)
0806 02:07:57.248553 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.248566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.248993 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.249545 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.249551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.250022 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:07:57.250673 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.250690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.251768 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.251772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.252896 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.252906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.253616 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.253619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.253720 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.254490 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:07:57.254505 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:07:57.257145 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:07:57.263821 (+ 6676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.263824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.263826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.264226 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.264600 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:07:57.264604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:07:57.264605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:07:57.264695 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:07:57.265182 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:07:57.265184 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:07:57.265265 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.265862 (+ 597us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.266010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":273,"spinlock_wait_cycles":20992,"thread_start_us":88,"threads_started":1}
I20260806 02:07:57.266786 4105 heartbeater.cc:347] Connected to a master server at 127.28.21.60:33711
I20260806 02:07:57.266886 4105 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.265105 4107 heartbeater.cc:464] Registering TS with master...
I20260806 02:07:57.267145 4105 heartbeater.cc:511] Master 127.28.21.60:33711 requested a full tablet report, sending...
I20260806 02:07:57.267145 4107 heartbeater.cc:511] Master 127.28.21.61:45155 requested a full tablet report, sending...
I20260806 02:07:57.267257 2806 ts_manager.cc:194] Re-registered known tserver with Master: a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383)
I20260806 02:07:57.267956 3436 ts_manager.cc:194] Registered new tserver with Master: a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383)
I20260806 02:07:57.267974 2806 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:41961
I20260806 02:07:57.268447 3630 ts_manager.cc:194] Registered new tserver with Master: a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383)
I20260806 02:07:57.272735 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.256745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.28.21.61:45155)
0806 02:07:57.257309 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.257313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.258264 (+ 951us) server_negotiation.cc:407] Connection header received
0806 02:07:57.258305 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.258311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.258363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.258479 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.259671 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.260531 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.262010 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.262637 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.262736 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.266113 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.266116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.266119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.266143 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.270821 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.271351 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.271356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.271406 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.271500 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.271822 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.271826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.271828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.272014 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.272159 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.272435 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.272564 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":100,"threads_started":1}
I20260806 02:07:57.273339 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.256993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.28.21.62:41525)
0806 02:07:57.257453 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.257467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.257480 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:57.257515 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.257517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.257568 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.257652 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.258825 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.259795 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.260988 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.261321 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.261417 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.269363 (+ 7946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.269367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.269369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.269394 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.271590 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.272080 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.272085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.272087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.272174 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.272482 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.272485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.272486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.272682 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.272815 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.273051 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.273172 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":93,"threads_started":1}
I20260806 02:07:57.273823 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.256866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.28.21.60:33711)
0806 02:07:57.257452 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.257457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.261803 (+ 4346us) server_negotiation.cc:407] Connection header received
0806 02:07:57.261975 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.261978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.262024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.262083 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.263118 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.265265 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.266740 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.267078 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.267153 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.270113 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.270117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.270119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.270151 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.271841 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.272323 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.272329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.272332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.272418 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.272725 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.272728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.272730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.272954 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.273080 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.273514 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.273660 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":110,"threads_started":1}
I20260806 02:07:57.276429 2806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:57.279532 2806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:57.282189 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: NON_VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: true } } }
I20260806 02:07:57.282353 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:57.286088 4124 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: attempting to promote NON_VOTER 2d1cc0a8dd354cc58016cb5844b44edb to VOTER
I20260806 02:07:57.287653 4124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } }
I20260806 02:07:57.289433 3642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 LEARNER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:07:57.289433 3450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Refusing update from remote peer 41c9b010f7f743c1a3d34c47562c7770: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:07:57.289901 4123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:07:57.290161 4123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:07:57.291821 4131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.292474 3450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.292393 3642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.293584 4124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.293694 4124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:57.293898 4124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.294008 4124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:57.294200 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.294308 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:57.294672 4124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.294777 4124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770 [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:57.295276 4134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41c9b010f7f743c1a3d34c47562c7770" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:57.295428 4134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:57.302563 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.297490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.28.21.1:40969)
0806 02:07:57.297874 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.297891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.297915 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:07:57.298002 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.298011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.298451 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.298658 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.298863 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.300141 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.301370 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.301948 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.302078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.302239 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.302292 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.302384 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":79,"threads_started":1}
I20260806 02:07:57.303030 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.297310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40969 (local address 127.0.0.1:40590)
0806 02:07:57.297455 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.297472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.297570 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.298628 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.298631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.298648 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.298738 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.298742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.300262 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.300265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.301249 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.301257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.302076 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.302079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.302163 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.302185 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.302249 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.302327 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260806 02:07:57.303750 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.297582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37723 (local address 127.0.0.1:45286)
0806 02:07:57.298426 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.298440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.298526 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.299442 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.299445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.299500 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.299598 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.299605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.300891 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.300895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.302084 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.302094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.303375 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.303378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.303468 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.303488 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.303565 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.303627 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":27,"thread_start_us":112,"threads_started":1}
I20260806 02:07:57.304028 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.298014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.28.21.2:37723)
0806 02:07:57.298981 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.298994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.299017 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:07:57.299090 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.299095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.299293 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.299444 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.299735 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.300772 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.302216 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.303060 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.303164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.303295 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.303615 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.303714 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":147,"threads_started":1}
I20260806 02:07:57.304442 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.298113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41383 (local address 127.0.0.1:35428)
0806 02:07:57.299660 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.299673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.299751 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.300394 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.300397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.300415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.300500 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.300507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.301798 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.301801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.302982 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.302994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.304063 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.304066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.304149 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.304169 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.304245 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.304314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260806 02:07:57.304491 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.298273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.28.21.3:41383)
0806 02:07:57.299984 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.299993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.300013 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:07:57.300077 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.300081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.300254 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.300414 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.300595 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.301665 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.303133 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.303967 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.304063 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.304191 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.304274 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.304358 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":93,"threads_started":1}
I20260806 02:07:57.306269 3749 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 (DEFAULT_TABLE table=test-table [id=df30cf46be9f4e198ebbb7f4e8c2baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:57.306723 4037 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 (DEFAULT_TABLE table=test-table [id=df30cf46be9f4e198ebbb7f4e8c2baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:57.306716 3891 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 (DEFAULT_TABLE table=test-table [id=df30cf46be9f4e198ebbb7f4e8c2baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:57.307497 3749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2bcf1713549708d41cee5bb4ab4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:57.307657 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2bcf1713549708d41cee5bb4ab4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:57.307785 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2bcf1713549708d41cee5bb4ab4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:57.314963 4140 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Bootstrap starting.
I20260806 02:07:57.316244 4141 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Bootstrap starting.
I20260806 02:07:57.317094 4140 tablet_bootstrap.cc:654] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:57.317852 4140 log.cc:824] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:57.318024 4141 tablet_bootstrap.cc:654] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:57.318750 4142 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Bootstrap starting.
I20260806 02:07:57.318902 4141 log.cc:824] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:57.319909 4140 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: No bootstrap required, opened a new log
I20260806 02:07:57.320114 4140 ts_tablet_manager.cc:1397] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:57.320887 4141 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: No bootstrap required, opened a new log
I20260806 02:07:57.320928 4142 tablet_bootstrap.cc:654] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:57.321077 4141 ts_tablet_manager.cc:1397] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:07:57.321720 4142 log.cc:824] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Log is configured to *not* fsync() on all Append() calls
I20260806 02:07:57.323691 4142 tablet_bootstrap.cc:492] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: No bootstrap required, opened a new log
I20260806 02:07:57.323875 4142 ts_tablet_manager.cc:1397] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:07:57.323933 4140 raft_consensus.cc:348] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.324203 4140 raft_consensus.cc:374] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:57.324262 4140 raft_consensus.cc:729] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15b8017a63e48ea87bee7e2965b08d7, State: Initialized, Role: FOLLOWER
I20260806 02:07:57.324753 4140 consensus_queue.cc:260] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.325605 4140 ts_tablet_manager.cc:1428] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:07:57.325625 4141 raft_consensus.cc:348] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.325874 4141 raft_consensus.cc:374] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:57.325934 4141 raft_consensus.cc:729] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6438ccb1f4724712b1785f7078d17c77, State: Initialized, Role: FOLLOWER
I20260806 02:07:57.326483 4141 consensus_queue.cc:260] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.326931 4106 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
I20260806 02:07:57.328636 4141 ts_tablet_manager.cc:1428] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:07:57.328519 4142 raft_consensus.cc:348] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.328754 4142 raft_consensus.cc:374] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:57.328797 4142 raft_consensus.cc:729] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b7fad6f52648348de44ad4e87a09ff, State: Initialized, Role: FOLLOWER
I20260806 02:07:57.328992 3819 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
I20260806 02:07:57.329267 4142 consensus_queue.cc:260] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.330199 4142 ts_tablet_manager.cc:1428] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:07:57.331322 3962 heartbeater.cc:503] Master 127.28.21.62:41525 was elected leader, sending a full tablet report...
W20260806 02:07:57.496793 4108 tablet.cc:2369] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:07:57.538643 3964 tablet.cc:2369] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:07:57.565366 3821 tablet.cc:2369] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:07:57.624416 4148 raft_consensus.cc:482] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:57.624629 4148 raft_consensus.cc:504] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.625801 4148 leader_election.cc:288] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969), a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:57.629603 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.626220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:49291 (local address 127.28.21.1:40969)
0806 02:07:57.626344 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.626349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.626364 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:07:57.626459 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.626462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.626516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.626609 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.626898 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.627401 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.628532 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.629007 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.629120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.629188 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.629386 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.629462 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:07:57.629634 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.626336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:60909 (local address 127.28.21.2:37723)
0806 02:07:57.626470 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.626474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.626486 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:07:57.626530 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.626532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.626572 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.626661 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:57.626955 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.627450 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.628576 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.629030 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.629146 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.629206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.629432 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.629503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:07:57.629647 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.626004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40969 (local address 127.28.21.3:49291)
0806 02:07:57.626157 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.626174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.626276 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.626619 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.626622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.626637 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.626784 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.626790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.627518 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.627523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.628394 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.628402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.629120 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.629124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.629222 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.629249 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.629356 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.629425 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":8}
I20260806 02:07:57.629652 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.626113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37723 (local address 127.28.21.3:60909)
0806 02:07:57.626257 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:57.626272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:57.626360 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:57.626674 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:57.626677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:57.626693 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:57.626836 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.626841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.627562 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.627568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.628417 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:57.628426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.629146 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.629150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:57.629278 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.629305 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:07:57.629397 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:07:57.629464 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":41}
I20260806 02:07:57.630340 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2bcf1713549708d41cee5bb4ab4b1" candidate_uuid: "a15b8017a63e48ea87bee7e2965b08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6438ccb1f4724712b1785f7078d17c77" is_pre_election: true
I20260806 02:07:57.630479 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2bcf1713549708d41cee5bb4ab4b1" candidate_uuid: "a15b8017a63e48ea87bee7e2965b08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b7fad6f52648348de44ad4e87a09ff" is_pre_election: true
I20260806 02:07:57.630716 3769 raft_consensus.cc:2454] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15b8017a63e48ea87bee7e2965b08d7 in term 0.
I20260806 02:07:57.630764 3912 raft_consensus.cc:2454] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a15b8017a63e48ea87bee7e2965b08d7 in term 0.
I20260806 02:07:57.631219 3989 leader_election.cc:302] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6438ccb1f4724712b1785f7078d17c77, a15b8017a63e48ea87bee7e2965b08d7, a8b7fad6f52648348de44ad4e87a09ff; no voters:
I20260806 02:07:57.631472 4148 raft_consensus.cc:2781] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:57.631554 4148 raft_consensus.cc:482] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:57.631597 4148 raft_consensus.cc:3037] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:57.632783 4148 raft_consensus.cc:504] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.633147 4148 leader_election.cc:288] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [CANDIDATE]: Term 1 election: Requested vote from peers 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969), a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723)
I20260806 02:07:57.633471 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2bcf1713549708d41cee5bb4ab4b1" candidate_uuid: "a15b8017a63e48ea87bee7e2965b08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b7fad6f52648348de44ad4e87a09ff"
I20260806 02:07:57.633488 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2bcf1713549708d41cee5bb4ab4b1" candidate_uuid: "a15b8017a63e48ea87bee7e2965b08d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6438ccb1f4724712b1785f7078d17c77"
I20260806 02:07:57.633607 3912 raft_consensus.cc:3037] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:57.633616 3769 raft_consensus.cc:3037] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:57.634673 3769 raft_consensus.cc:2454] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15b8017a63e48ea87bee7e2965b08d7 in term 1.
I20260806 02:07:57.634706 3912 raft_consensus.cc:2454] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a15b8017a63e48ea87bee7e2965b08d7 in term 1.
I20260806 02:07:57.634970 3989 leader_election.cc:302] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15b8017a63e48ea87bee7e2965b08d7, a8b7fad6f52648348de44ad4e87a09ff; no voters:
I20260806 02:07:57.635150 4148 raft_consensus.cc:2781] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:57.635411 4148 raft_consensus.cc:686] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 LEADER]: Becoming Leader. State: Replica: a15b8017a63e48ea87bee7e2965b08d7, State: Running, Role: LEADER
I20260806 02:07:57.635797 4148 consensus_queue.cc:237] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } }
I20260806 02:07:57.639279 2805 catalog_manager.cc:5495] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "a15b8017a63e48ea87bee7e2965b08d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } health_report { overall_health: HEALTHY } } }
I20260806 02:07:57.672724 4139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.669363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.28.21.3:41383)
0806 02:07:57.669527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.669530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.669543 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:57.669619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.669622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.669675 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.669763 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:57.670013 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.670587 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.671380 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.671617 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.671695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.671961 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:57.672373 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:57.672498 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.672564 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.672614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:07:57.678503 3769 raft_consensus.cc:1261] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Refusing update from remote peer a15b8017a63e48ea87bee7e2965b08d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:57.678514 3912 raft_consensus.cc:1261] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Refusing update from remote peer a15b8017a63e48ea87bee7e2965b08d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:57.679168 4151 consensus_queue.cc:1035] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:57.679354 4148 consensus_queue.cc:1035] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:57.688759 4156 mvcc.cc:205] Tried to move back new op lower bound from 7315382590164017152 to 7315382589998596096. Current Snapshot: MvccSnapshot[applied={T|T < 7315382590164017152}]
I20260806 02:07:57.699738 2806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:57.699916 2806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:57.704301 2806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41c9b010f7f743c1a3d34c47562c7770: Sending DeleteTablet for 3 replicas of tablet f1a2bcf1713549708d41cee5bb4ab4b1
I20260806 02:07:57.705808 4037 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:57 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260806 02:07:57.705771 3749 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:57 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:40590
I20260806 02:07:57.706632 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a2bcf1713549708d41cee5bb4ab4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:57 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:45286
I20260806 02:07:57.707059 4165 tablet_replica.cc:330] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: stopping tablet replica
I20260806 02:07:57.707479 4165 raft_consensus.cc:2229] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:57.708930 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 2771
I20260806 02:07:57.712007 4165 raft_consensus.cc:2258] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:57.713845 4165 ts_tablet_manager.cc:1905] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:57.718112 4165 ts_tablet_manager.cc:1918] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:57.718246 4165 log.cc:1196] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/f1a2bcf1713549708d41cee5bb4ab4b1
I20260806 02:07:57.718675 4165 ts_tablet_manager.cc:1939] T f1a2bcf1713549708d41cee5bb4ab4b1 P a15b8017a63e48ea87bee7e2965b08d7: Deleting consensus metadata
I20260806 02:07:57.720227 4168 tablet_replica.cc:330] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: stopping tablet replica
W20260806 02:07:57.720367 3989 connection.cc:535] server connection from 127.0.0.1:35428 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:07:57.708431 4166 tablet_replica.cc:330] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: stopping tablet replica
I20260806 02:07:57.720880 4166 raft_consensus.cc:2229] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:57.720920 4168 raft_consensus.cc:2229] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:57.721277 4168 raft_consensus.cc:2258] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:57.723474 4168 ts_tablet_manager.cc:1905] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:57.724969 4166 raft_consensus.cc:2258] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:57.728315 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.707466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.28.21.61:45155)
0806 02:07:57.707612 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.707617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.708095 (+ 478us) server_negotiation.cc:407] Connection header received
0806 02:07:57.708241 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.708244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.708300 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.708382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.718522 (+ 10140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.719382 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.720681 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.721058 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.721157 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.723531 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.723539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.723546 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.723579 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.725570 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.726158 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.726162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.726164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.726258 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.726636 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.726640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.726642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.726883 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.727016 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.727982 (+ 966us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.728135 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:07:57.732872 4168 ts_tablet_manager.cc:1918] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:57.734269 4166 ts_tablet_manager.cc:1905] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:57.737759 4168 log.cc:1196] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/f1a2bcf1713549708d41cee5bb4ab4b1
I20260806 02:07:57.738268 4168 ts_tablet_manager.cc:1939] T f1a2bcf1713549708d41cee5bb4ab4b1 P a8b7fad6f52648348de44ad4e87a09ff: Deleting consensus metadata
I20260806 02:07:57.738732 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.707517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.28.21.60:33711)
0806 02:07:57.707637 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.707641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.727742 (+ 20101us) server_negotiation.cc:407] Connection header received
0806 02:07:57.727902 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.727905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.727981 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.728059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.729372 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.730231 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.731506 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.731893 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.731980 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.733903 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.733907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.733910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.733941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.735924 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.736490 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.736493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.736495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.736594 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.737243 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.737247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.737249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.737500 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.737646 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.738380 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.738565 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 02:07:57.740263 3843 connection.cc:441] server connection from 127.0.0.1:45286 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45286 (request call id 1) could send its response
I20260806 02:07:57.740722 4166 ts_tablet_manager.cc:1918] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:57.740921 4166 log.cc:1196] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/f1a2bcf1713549708d41cee5bb4ab4b1
I20260806 02:07:57.741459 4166 ts_tablet_manager.cc:1939] T f1a2bcf1713549708d41cee5bb4ab4b1 P 6438ccb1f4724712b1785f7078d17c77: Deleting consensus metadata
I20260806 02:07:57.741734 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.731219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.28.21.61:45155)
0806 02:07:57.731669 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.731673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.732929 (+ 1256us) server_negotiation.cc:407] Connection header received
0806 02:07:57.733112 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.733115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.733154 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.733204 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.734047 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.734845 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.736004 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.736337 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.736405 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.738116 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.738120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.738123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.738147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.739954 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.740418 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.740421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.740423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.740505 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.740803 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.740806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.740808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.741019 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.741115 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.741463 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.741579 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
W20260806 02:07:57.743002 3699 connection.cc:441] server connection from 127.0.0.1:40590 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40590 (request call id 1) could send its response
I20260806 02:07:57.748181 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:57.732172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.28.21.60:33711)
0806 02:07:57.732278 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:57.732282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:57.739298 (+ 7016us) server_negotiation.cc:407] Connection header received
0806 02:07:57.739476 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:57.739480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:57.739525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:57.739576 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:07:57.740682 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.741532 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.742900 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:57.743252 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:57.743338 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:57.744778 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:07:57.744782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:07:57.744785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:07:57.744809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:07:57.746481 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.746924 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.746928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.746929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.747026 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:07:57.747356 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:07:57.747359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:07:57.747361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:07:57.747569 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:07:57.747698 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:57.747899 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:07:57.748025 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:07:58.691594 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:58.690897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:46867)
0806 02:07:58.691290 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:58.691411 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
W20260806 02:07:58.692044 3819 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:41525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:58.692703 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:58.692122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:50953)
0806 02:07:58.692484 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:58.692593 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
W20260806 02:07:58.693100 4106 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:41525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:58.693257 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:58.692680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:50653)
0806 02:07:58.693067 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:58.693152 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":69,"threads_started":1}
W20260806 02:07:58.693643 3962 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:41525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:59.205114 4179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41c9b010f7f743c1a3d34c47562c7770)
I20260806 02:07:59.205318 4179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } }
I20260806 02:07:59.205755 4180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41c9b010f7f743c1a3d34c47562c7770)
I20260806 02:07:59.205945 4180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } }
I20260806 02:07:59.206911 4180 leader_election.cc:288] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525), 2d1cc0a8dd354cc58016cb5844b44edb (127.28.21.60:33711)
I20260806 02:07:59.207291 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.206642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45476)
0806 02:07:59.207084 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.207174 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":60,"threads_started":1}
I20260806 02:07:59.207475 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.207025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45482)
0806 02:07:59.207286 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.207384 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260806 02:07:59.207600 3614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
W20260806 02:07:59.207638 3424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:59.207862 4179 leader_election.cc:288] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525), 90286d46c56947e9a44bbb92505233c1 (127.28.21.61:45155)
I20260806 02:07:59.208534 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.208298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45494)
0806 02:07:59.208413 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.208473 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:07:59.208709 3614 leader_election.cc:334] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525): Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:59.209322 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.209079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45502)
0806 02:07:59.209188 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.209239 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:07:59.209422 3424 leader_election.cc:334] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525): Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:59.212103 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.207443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:45155 (local address 127.0.0.1:59082)
0806 02:07:59.208082 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.208095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.208187 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.208598 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.208602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.208620 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.208719 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.208724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.209724 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.209729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.210901 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.210911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.211789 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.211794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.211880 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.211900 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.211963 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.212016 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":68,"threads_started":1}
I20260806 02:07:59.212260 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.207709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.28.21.61:45155)
0806 02:07:59.207965 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.207971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.208275 (+ 304us) server_negotiation.cc:407] Connection header received
0806 02:07:59.208349 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.208352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.208398 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.208475 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.208831 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.209616 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.211026 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.211659 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.211772 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.211833 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.212006 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.212084 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260806 02:07:59.212671 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.207096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:33711 (local address 127.0.0.1:56602)
0806 02:07:59.207635 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.207651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.207747 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.208858 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.208862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.208900 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.209231 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.209236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.210171 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.210174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.211319 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.211328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.212264 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.212271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.212398 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.212425 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.212480 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.212554 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":67,"threads_started":1}
I20260806 02:07:59.212648 3450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90286d46c56947e9a44bbb92505233c1" is_pre_election: true
I20260806 02:07:59.212852 3450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d1cc0a8dd354cc58016cb5844b44edb in term 1.
I20260806 02:07:59.213230 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.208024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.28.21.60:33711)
0806 02:07:59.208577 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.208587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.208604 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:07:59.208656 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.208663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.208711 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.209344 (+ 633us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.209486 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.210061 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.211429 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.212117 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.212881 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.212981 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.213020 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.213114 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":112,"threads_started":1}
I20260806 02:07:59.213218 3617 leader_election.cc:302] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d1cc0a8dd354cc58016cb5844b44edb, 90286d46c56947e9a44bbb92505233c1; no voters: 41c9b010f7f743c1a3d34c47562c7770
I20260806 02:07:59.213449 4179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:07:59.213661 3642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90286d46c56947e9a44bbb92505233c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" is_pre_election: true
I20260806 02:07:59.213506 4179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41c9b010f7f743c1a3d34c47562c7770)
I20260806 02:07:59.213965 4179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:07:59.215157 4179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } }
I20260806 02:07:59.215446 3642 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90286d46c56947e9a44bbb92505233c1 in current term 2: Already voted for candidate 2d1cc0a8dd354cc58016cb5844b44edb in this term.
I20260806 02:07:59.215591 4179 leader_election.cc:288] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 election: Requested vote from peers 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525), 90286d46c56947e9a44bbb92505233c1 (127.28.21.61:45155)
I20260806 02:07:59.215984 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.215671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45506)
0806 02:07:59.215890 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.215934 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 02:07:59.216081 3450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90286d46c56947e9a44bbb92505233c1"
I20260806 02:07:59.216192 3450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:07:59.216493 3427 leader_election.cc:302] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90286d46c56947e9a44bbb92505233c1; no voters: 2d1cc0a8dd354cc58016cb5844b44edb, 41c9b010f7f743c1a3d34c47562c7770
I20260806 02:07:59.216769 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.216511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45520)
0806 02:07:59.216666 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.216715 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:07:59.216884 3614 leader_election.cc:334] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525): Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
I20260806 02:07:59.217389 3450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d1cc0a8dd354cc58016cb5844b44edb in term 2.
I20260806 02:07:59.217512 4180 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:07:59.217706 3617 leader_election.cc:302] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d1cc0a8dd354cc58016cb5844b44edb, 90286d46c56947e9a44bbb92505233c1; no voters: 41c9b010f7f743c1a3d34c47562c7770
I20260806 02:07:59.217890 4179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:07:59.218087 4179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [term 2 LEADER]: Becoming Leader. State: Replica: 2d1cc0a8dd354cc58016cb5844b44edb, State: Running, Role: LEADER
I20260806 02:07:59.218341 4179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } }
I20260806 02:07:59.219688 4190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1cc0a8dd354cc58016cb5844b44edb. Latest consensus state: current_term: 2 leader_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:59.219807 4190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:59.220091 4192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:07:59.222368 4192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:07:59.222810 4192 catalog_manager.cc:1259] Loaded cluster ID: 39cbdd7d61684832a5faf54edf9cfc2f
I20260806 02:07:59.222862 4192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:07:59.223413 4192 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:07:59.223801 4192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Loaded TSK: 0
I20260806 02:07:59.224493 4192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:07:59.274303 3629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:07:59.277459 3629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:07:59.280380 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.280138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45536)
0806 02:07:59.280258 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.280315 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 02:07:59.280453 3450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [term 2 FOLLOWER]: Refusing update from remote peer 2d1cc0a8dd354cc58016cb5844b44edb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 02:07:59.281064 4190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 02:07:59.281217 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.280966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.0.0.1:45548)
0806 02:07:59.281102 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.281152 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:07:59.281391 3614 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb -> Peer 41c9b010f7f743c1a3d34c47562c7770 (127.28.21.62:41525): Couldn't send request to peer 41c9b010f7f743c1a3d34c47562c7770. Status: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:07:59.282678 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1cc0a8dd354cc58016cb5844b44edb. Latest consensus state: current_term: 2 leader_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:59.282814 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:59.283826 4190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:59.283943 4190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb [sys.catalog]: This master's current role is: LEADER
I20260806 02:07:59.284132 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41c9b010f7f743c1a3d34c47562c7770" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41525 } } peers { permanent_uuid: "90286d46c56947e9a44bbb92505233c1" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "2d1cc0a8dd354cc58016cb5844b44edb" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 33711 } attrs { promote: false } } }
I20260806 02:07:59.284253 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90286d46c56947e9a44bbb92505233c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:07:59.294966 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.290221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41383 (local address 127.0.0.1:35440)
0806 02:07:59.290675 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.290691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.290778 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.291053 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.291056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.291076 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.291200 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.291207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.292327 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.292331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.293624 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.293634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.294553 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.294556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.294655 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.294694 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.294745 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.294821 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":1}
I20260806 02:07:59.295473 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.290344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.28.21.3:41383)
0806 02:07:59.290789 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.290793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.290809 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:07:59.290866 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.290869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.290932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.291322 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.291451 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.292196 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.293776 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.294378 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.295159 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.295223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.295264 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.295348 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":99,"threads_started":1}
I20260806 02:07:59.296067 4037 tablet_service.cc:1467] Processing CreateTablet for tablet 36084843fea24b1ea8548a2063271600 (DEFAULT_TABLE table=test-table [id=7284af1207d841678e555ea960b2585d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:59.296239 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.290465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40969 (local address 127.0.0.1:40606)
0806 02:07:59.290784 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.290799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.290879 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.291610 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.291613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.291632 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.291721 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.291729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.292786 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.292790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.294905 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.294913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.295878 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.295881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.295968 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.295990 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.296058 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.296136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260806 02:07:59.296387 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36084843fea24b1ea8548a2063271600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:59.296649 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.290657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.28.21.1:40969)
0806 02:07:59.291320 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.291324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.291337 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:07:59.291390 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.291392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.291479 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.291832 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.291952 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.292662 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.295018 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.295625 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.295724 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.295818 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.296439 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.296527 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":79,"threads_started":1}
I20260806 02:07:59.297070 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.291023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37723 (local address 127.0.0.1:45294)
0806 02:07:59.292334 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.292347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.292469 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.293241 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.293244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.293262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.293353 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.293359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.294385 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.294388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.295579 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.295590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.296631 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.296637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.296752 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.296786 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.296852 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.296920 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":94,"threads_started":1}
I20260806 02:07:59.297254 3749 tablet_service.cc:1467] Processing CreateTablet for tablet 36084843fea24b1ea8548a2063271600 (DEFAULT_TABLE table=test-table [id=7284af1207d841678e555ea960b2585d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:59.297413 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.291145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.28.21.2:37723)
0806 02:07:59.292914 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.292919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.292934 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:07:59.292988 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.292992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.293038 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.293125 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.293503 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.294244 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.295695 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.296376 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.296493 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.296574 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.297207 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.297294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":80,"threads_started":1}
I20260806 02:07:59.297518 3749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36084843fea24b1ea8548a2063271600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:59.299144 4202 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Bootstrap starting.
I20260806 02:07:59.299970 4202 tablet_bootstrap.cc:654] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:59.299999 4203 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: Bootstrap starting.
I20260806 02:07:59.300186 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 36084843fea24b1ea8548a2063271600 (DEFAULT_TABLE table=test-table [id=7284af1207d841678e555ea960b2585d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:07:59.300454 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36084843fea24b1ea8548a2063271600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:07:59.300887 4203 tablet_bootstrap.cc:654] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:59.301218 4202 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: No bootstrap required, opened a new log
I20260806 02:07:59.301323 4202 ts_tablet_manager.cc:1397] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:07:59.301786 4202 raft_consensus.cc:348] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.301935 4202 raft_consensus.cc:374] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:59.301975 4202 raft_consensus.cc:729] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15b8017a63e48ea87bee7e2965b08d7, State: Initialized, Role: FOLLOWER
I20260806 02:07:59.302102 4202 consensus_queue.cc:260] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.302481 4202 ts_tablet_manager.cc:1428] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:07:59.303254 4207 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Bootstrap starting.
I20260806 02:07:59.303539 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.303129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:45529)
0806 02:07:59.303361 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.303428 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260806 02:07:59.304028 4203 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: No bootstrap required, opened a new log
I20260806 02:07:59.304114 4203 ts_tablet_manager.cc:1397] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 02:07:59.304179 4207 tablet_bootstrap.cc:654] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Neither blocks nor log segments found. Creating new log.
I20260806 02:07:59.304615 4203 raft_consensus.cc:348] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.304741 4203 raft_consensus.cc:374] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:59.304777 4203 raft_consensus.cc:729] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6438ccb1f4724712b1785f7078d17c77, State: Initialized, Role: FOLLOWER
I20260806 02:07:59.304910 4203 consensus_queue.cc:260] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.305253 4203 ts_tablet_manager.cc:1428] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:59.305284 4105 heartbeater.cc:503] Master 127.28.21.60:33711 was elected leader, sending a full tablet report...
I20260806 02:07:59.306499 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.306210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:60387)
0806 02:07:59.306342 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.306397 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:07:59.307578 3818 heartbeater.cc:503] Master 127.28.21.60:33711 was elected leader, sending a full tablet report...
I20260806 02:07:59.307719 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.305848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:46423)
0806 02:07:59.307564 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.307621 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1655,"thread_start_us":104,"threads_started":1}
I20260806 02:07:59.308207 4207 tablet_bootstrap.cc:492] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: No bootstrap required, opened a new log
I20260806 02:07:59.308306 4207 ts_tablet_manager.cc:1397] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:07:59.308808 4207 raft_consensus.cc:348] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.308915 4207 raft_consensus.cc:374] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:07:59.308943 4207 raft_consensus.cc:729] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b7fad6f52648348de44ad4e87a09ff, State: Initialized, Role: FOLLOWER
I20260806 02:07:59.309032 4207 consensus_queue.cc:260] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.309266 4207 ts_tablet_manager.cc:1428] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:07:59.309978 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.309577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:60075)
0806 02:07:59.309823 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.309878 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260806 02:07:59.310678 3961 heartbeater.cc:503] Master 127.28.21.60:33711 was elected leader, sending a full tablet report...
I20260806 02:07:59.310811 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.310399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:41713)
0806 02:07:59.310642 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.310711 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260806 02:07:59.343456 4213 raft_consensus.cc:482] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:07:59.343648 4213 raft_consensus.cc:504] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.346680 4213 leader_election.cc:288] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383), 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:59.352671 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.347428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:60203 (local address 127.28.21.1:40969)
0806 02:07:59.347550 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.347555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.347570 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:07:59.347741 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.347747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.347817 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.347872 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.348426 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.349356 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.350935 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.351677 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.351813 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.351901 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.352371 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.352498 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:07:59.352802 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.347034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:45439 (local address 127.28.21.3:41383)
0806 02:07:59.347206 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.347212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.347438 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:07:59.347733 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.347736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.347798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.347849 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:07:59.348487 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.349330 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.351498 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.352153 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.352225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.352282 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.352606 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.352688 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 02:07:59.353066 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.347166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40969 (local address 127.28.21.2:60203)
0806 02:07:59.347478 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.347494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.347589 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.348046 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.348051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.348076 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.348283 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.348294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.349474 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.349477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.350780 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.350793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.352025 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.352029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.352195 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.352230 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.352812 (+ 582us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.352906 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":53,"spinlock_wait_cycles":10240}
I20260806 02:07:59.353232 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.346882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41383 (local address 127.28.21.2:45439)
0806 02:07:59.347318 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.347334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:07:59.347552 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:07:59.348098 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:07:59.348103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:07:59.348132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:07:59.348298 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.348304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.349543 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.349546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.351357 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:07:59.351367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.352368 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.352371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:07:59.352478 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.352499 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:07:59.352970 (+ 471us) client_negotiation.cc:240] Negotiation successful
0806 02:07:59.353042 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":773,"spinlock_wait_cycles":768}
I20260806 02:07:59.353758 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36084843fea24b1ea8548a2063271600" candidate_uuid: "a8b7fad6f52648348de44ad4e87a09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15b8017a63e48ea87bee7e2965b08d7" is_pre_election: true
I20260806 02:07:59.353976 4057 raft_consensus.cc:2454] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b7fad6f52648348de44ad4e87a09ff in term 0.
I20260806 02:07:59.354429 3843 leader_election.cc:302] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15b8017a63e48ea87bee7e2965b08d7, a8b7fad6f52648348de44ad4e87a09ff; no voters:
I20260806 02:07:59.354668 4213 raft_consensus.cc:2781] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:07:59.354710 4213 raft_consensus.cc:482] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:07:59.354743 4213 raft_consensus.cc:3037] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:59.354736 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36084843fea24b1ea8548a2063271600" candidate_uuid: "a8b7fad6f52648348de44ad4e87a09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6438ccb1f4724712b1785f7078d17c77" is_pre_election: true
I20260806 02:07:59.354872 3769 raft_consensus.cc:2454] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b7fad6f52648348de44ad4e87a09ff in term 0.
I20260806 02:07:59.355973 4213 raft_consensus.cc:504] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.356297 4213 leader_election.cc:288] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [CANDIDATE]: Term 1 election: Requested vote from peers a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383), 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969)
I20260806 02:07:59.356701 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36084843fea24b1ea8548a2063271600" candidate_uuid: "a8b7fad6f52648348de44ad4e87a09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15b8017a63e48ea87bee7e2965b08d7"
I20260806 02:07:59.356817 4057 raft_consensus.cc:3037] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:59.356861 3769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36084843fea24b1ea8548a2063271600" candidate_uuid: "a8b7fad6f52648348de44ad4e87a09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6438ccb1f4724712b1785f7078d17c77"
I20260806 02:07:59.356997 3769 raft_consensus.cc:3037] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:07:59.357910 4057 raft_consensus.cc:2454] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b7fad6f52648348de44ad4e87a09ff in term 1.
I20260806 02:07:59.358183 3843 leader_election.cc:302] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a15b8017a63e48ea87bee7e2965b08d7, a8b7fad6f52648348de44ad4e87a09ff; no voters:
I20260806 02:07:59.358322 4213 raft_consensus.cc:2781] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:07:59.358347 3769 raft_consensus.cc:2454] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b7fad6f52648348de44ad4e87a09ff in term 1.
I20260806 02:07:59.358533 4213 raft_consensus.cc:686] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 1 LEADER]: Becoming Leader. State: Replica: a8b7fad6f52648348de44ad4e87a09ff, State: Running, Role: LEADER
I20260806 02:07:59.358762 4213 consensus_queue.cc:237] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } }
I20260806 02:07:59.359723 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.359086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:35141)
0806 02:07:59.359605 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.359660 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429}
I20260806 02:07:59.361291 3629 catalog_manager.cc:5495] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2). New cstate: current_term: 1 leader_uuid: "a8b7fad6f52648348de44ad4e87a09ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b7fad6f52648348de44ad4e87a09ff" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37723 } health_report { overall_health: HEALTHY } } }
I20260806 02:07:59.373668 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.370056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.28.21.2:37723)
0806 02:07:59.370174 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:07:59.370180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:07:59.370213 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:07:59.370395 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:07:59.370468 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:07:59.370539 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:07:59.370649 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:07:59.370887 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.371379 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.372266 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:07:59.372500 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:07:59.372604 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:07:59.372869 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:07:59.373297 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:07:59.373418 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:07:59.373490 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:07:59.373541 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:07:59.376474 3769 raft_consensus.cc:1261] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Refusing update from remote peer a8b7fad6f52648348de44ad4e87a09ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:59.376989 4217 consensus_queue.cc:1035] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "6438ccb1f4724712b1785f7078d17c77" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:59.377146 4057 raft_consensus.cc:1261] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Refusing update from remote peer a8b7fad6f52648348de44ad4e87a09ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:07:59.377666 4217 consensus_queue.cc:1035] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15b8017a63e48ea87bee7e2965b08d7" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:07:59.378866 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.378591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:55621)
0806 02:07:59.378736 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.378790 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:07:59.379981 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.379577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:37107)
0806 02:07:59.379825 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.379890 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260806 02:07:59.381500 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.381221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:60969)
0806 02:07:59.381365 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.381417 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 02:07:59.382719 4221 mvcc.cc:205] Tried to move back new op lower bound from 7315382597120716800 to 7315382597055639552. Current Snapshot: MvccSnapshot[applied={T|T < 7315382597120716800}]
I20260806 02:07:59.383589 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.383336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.1:47887)
0806 02:07:59.383475 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.383524 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 02:07:59.383852 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.383188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.2:50139)
0806 02:07:59.383742 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.383793 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489}
I20260806 02:07:59.384169 4223 mvcc.cc:205] Tried to move back new op lower bound from 7315382597120716800 to 7315382597055639552. Current Snapshot: MvccSnapshot[applied={T|T < 7315382597120716800 or (T in {7315382597120716800})}]
I20260806 02:07:59.384725 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.384461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41525 (local address 127.28.21.3:56597)
0806 02:07:59.384594 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.384655 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:07:59.391996 3630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:59.392155 3630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:07:59.395715 3630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d1cc0a8dd354cc58016cb5844b44edb: Sending DeleteTablet for 3 replicas of tablet 36084843fea24b1ea8548a2063271600
I20260806 02:07:59.396469 4037 tablet_service.cc:1514] Processing DeleteTablet for tablet 36084843fea24b1ea8548a2063271600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:59 UTC) from {username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:35440
I20260806 02:07:59.396531 3749 tablet_service.cc:1514] Processing DeleteTablet for tablet 36084843fea24b1ea8548a2063271600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:59 UTC) from {username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:40606
I20260806 02:07:59.396842 4230 tablet_replica.cc:330] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: stopping tablet replica
I20260806 02:07:59.396862 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 36084843fea24b1ea8548a2063271600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:07:59 UTC) from {username='oryx', principal='oryx/127.28.21.60@KRBTEST.COM'} at 127.0.0.1:45294
I20260806 02:07:59.397094 4230 raft_consensus.cc:2229] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:59.397208 4232 tablet_replica.cc:330] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: stopping tablet replica
I20260806 02:07:59.397324 4230 raft_consensus.cc:2258] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:59.397513 4231 tablet_replica.cc:330] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77: stopping tablet replica
I20260806 02:07:59.397627 4231 raft_consensus.cc:2229] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:07:59.397756 4231 raft_consensus.cc:2258] T 36084843fea24b1ea8548a2063271600 P 6438ccb1f4724712b1785f7078d17c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:59.398118 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3590
I20260806 02:07:59.398346 4230 ts_tablet_manager.cc:1905] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:59.400947 4230 ts_tablet_manager.cc:1918] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:59.401046 4230 log.cc:1196] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/36084843fea24b1ea8548a2063271600
I20260806 02:07:59.401316 4230 ts_tablet_manager.cc:1939] T 36084843fea24b1ea8548a2063271600 P a15b8017a63e48ea87bee7e2965b08d7: Deleting consensus metadata
I20260806 02:07:59.401993 3615 catalog_manager.cc:4841] TS a15b8017a63e48ea87bee7e2965b08d7 (127.28.21.3:41383): tablet 36084843fea24b1ea8548a2063271600 (table test-table [id=7284af1207d841678e555ea960b2585d]) successfully deleted
I20260806 02:07:59.397341 4232 raft_consensus.cc:2229] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:07:59.405025 4232 raft_consensus.cc:2258] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:07:59.405746 4232 ts_tablet_manager.cc:1905] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:07:59.408007 4232 ts_tablet_manager.cc:1918] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:07:59.408102 4232 log.cc:1196] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/36084843fea24b1ea8548a2063271600
I20260806 02:07:59.408416 4232 ts_tablet_manager.cc:1939] T 36084843fea24b1ea8548a2063271600 P a8b7fad6f52648348de44ad4e87a09ff: Deleting consensus metadata
I20260806 02:07:59.409046 3617 catalog_manager.cc:4841] TS a8b7fad6f52648348de44ad4e87a09ff (127.28.21.2:37723): tablet 36084843fea24b1ea8548a2063271600 (table test-table [id=7284af1207d841678e555ea960b2585d]) successfully deleted
I20260806 02:07:59.416945 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3826
I20260806 02:07:59.417174 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:07:59.416780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40969 (local address 127.0.0.1:40608)
0806 02:07:59.417005 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 02:07:59.417071 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260806 02:07:59.418440 3615 catalog_manager.cc:4584] TS 6438ccb1f4724712b1785f7078d17c77 (127.28.21.1:40969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36084843fea24b1ea8548a2063271600: Network error: Client connection negotiation failed: client connection to 127.28.21.1:40969: connect: Connection refused (error 111)
I20260806 02:07:59.424178 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3968
I20260806 02:07:59.432137 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3293
I20260806 02:07:59.438766 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 3405
2026-08-06T02:07:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5634 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:07:59 dist-test-slave-gp6n krb5kdc[4235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:07:59 dist-test-slave-gp6n krb5kdc[4235](info): set up 2 sockets
Aug 06 02:07:59 dist-test-slave-gp6n krb5kdc[4235](info): commencing operation
krb5kdc: starting...
W20260806 02:08:00.533280 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:08:00 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab.
Entry for principal oryx/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab.
I20260806 02:08:00.663523 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43607
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab
--principal=oryx/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:00.775321 4250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:00.775611 4250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:00.775682 4250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:00.779610 4250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:00.779704 4250 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:08:00.779737 4250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:00.779780 4250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:00.779819 4250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:00.779855 4250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:00.785039 4250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:43607
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.62.keytab
--principal=oryx/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:00.786092 4250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:00.787135 4250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:00.795781 4256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:00.795794 4255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:00.795898 4250 server_base.cc:1034] running on GCE node
W20260806 02:08:00.795789 4258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:00.796658 4250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:00.797789 4250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:00.798977 4250 hybrid_clock.cc:648] HybridClock initialized: now 1785982080798961 us; error 26 us; skew 500 ppm
Aug 06 02:08:00 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982080, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:00.801712 4250 init.cc:377] Logged in from keytab as oryx/127.28.21.62@KRBTEST.COM (short username oryx)
I20260806 02:08:00.802778 4250 webserver.cc:460] Webserver started at http://127.28.21.62:33903/ using document root <none> and password file <none>
I20260806 02:08:00.803340 4250 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:00.803434 4250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:00.803673 4250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:00.805716 4250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "49167c7be87f4ca4b9d3a8d5111a6da3"
format_stamp: "Formatted at 2026-08-06 02:08:00 on dist-test-slave-gp6n"
server_key: "b6012db514dfd0dc3a255ef982619472"
server_key_iv: "5b8eff3db2c3fc8d41bfdde3624e3172"
server_key_version: "encryptionkey@0"
I20260806 02:08:00.806298 4250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "49167c7be87f4ca4b9d3a8d5111a6da3"
format_stamp: "Formatted at 2026-08-06 02:08:00 on dist-test-slave-gp6n"
server_key: "b6012db514dfd0dc3a255ef982619472"
server_key_iv: "5b8eff3db2c3fc8d41bfdde3624e3172"
server_key_version: "encryptionkey@0"
I20260806 02:08:00.810034 4250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:08:00.812501 4265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:00.813735 4250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:00.813877 4250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "49167c7be87f4ca4b9d3a8d5111a6da3"
format_stamp: "Formatted at 2026-08-06 02:08:00 on dist-test-slave-gp6n"
server_key: "b6012db514dfd0dc3a255ef982619472"
server_key_iv: "5b8eff3db2c3fc8d41bfdde3624e3172"
server_key_version: "encryptionkey@0"
I20260806 02:08:00.813992 4250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:00.827483 4250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:00.830828 4250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:00.831056 4250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:00.839393 4250 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:43607
I20260806 02:08:00.839407 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:43607 every 8 connection(s)
I20260806 02:08:00.840389 4250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:00.843180 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:00.848697 4332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Bootstrap starting.
I20260806 02:08:00.850556 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 4250
I20260806 02:08:00.850696 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:08:00.851009 28756 external_mini_cluster.cc:1428] Setting key 9c2b079f3ef5faf6100f74d3a84bbe58
I20260806 02:08:00.851509 4332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:00.852415 4332 log.cc:824] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:00.854367 4332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: No bootstrap required, opened a new log
I20260806 02:08:00.857803 4332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } }
I20260806 02:08:00.858098 4332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:00.858151 4332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49167c7be87f4ca4b9d3a8d5111a6da3, State: Initialized, Role: FOLLOWER
I20260806 02:08:00.858670 4332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } }
I20260806 02:08:00.858804 4332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:00.858894 4332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:00.859014 4332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:08:00 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:00.860172 4332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } }
I20260806 02:08:00.860683 4332 leader_election.cc:302] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49167c7be87f4ca4b9d3a8d5111a6da3; no voters:
I20260806 02:08:00.861167 4332 leader_election.cc:288] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:08:00.861289 4337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:00.861584 4337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [term 1 LEADER]: Becoming Leader. State: Replica: 49167c7be87f4ca4b9d3a8d5111a6da3, State: Running, Role: LEADER
I20260806 02:08:00.861925 4337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } }
I20260806 02:08:00.862773 4332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:00.863744 4339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49167c7be87f4ca4b9d3a8d5111a6da3. Latest consensus state: current_term: 1 leader_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } } }
I20260806 02:08:00.863880 4339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:00.864225 4338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49167c7be87f4ca4b9d3a8d5111a6da3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 43607 } } }
I20260806 02:08:00.864363 4338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:00.866954 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:00.852176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.28.21.62:43607)
0806 02:08:00.852669 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:00.852686 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:00.852723 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:08:00.853568 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:00.853588 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:00.853915 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:00.854262 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:00.855131 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:00.856158 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:00.857043 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:00.857532 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:00.857704 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:00.860972 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:00.860978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:00.860993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:00.861022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:00.863156 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:00.863766 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:00.863772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:00.863778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:00.863923 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:00.864310 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:00.864314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:00.864315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:00.864925 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:00.865115 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:00.865760 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 02:08:00.866015 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":137,"threads_started":1}
W20260806 02:08:00.867872 4352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:08:00.867966 4352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:08:00.868048 4346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:00.868875 4346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:00.874795 4346 catalog_manager.cc:1347] Generated new cluster ID: 0a53c9f2ffcb41109148777c1ae4c158
I20260806 02:08:00.874862 4346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:00.885860 4346 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:08:00.887163 4346 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:00.894037 4346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Generated new TSK 0
I20260806 02:08:00.894862 4346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
I20260806 02:08:00.925745 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43607
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:01.038753 4359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:01.039060 4359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:01.039139 4359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:01.043218 4359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:01.043301 4359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:01.043375 4359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:08:01.043463 4359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:01.048857 4359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:01.050019 4359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:01.051123 4359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:01.061062 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:01.061071 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:01.061071 4367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:01.061748 4359 server_base.cc:1034] running on GCE node
I20260806 02:08:01.062180 4359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:01.062884 4359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:01.064085 4359 hybrid_clock.cc:648] HybridClock initialized: now 1785982081064050 us; error 45 us; skew 500 ppm
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:01.067036 4359 init.cc:377] Logged in from keytab as oryx/127.28.21.1@KRBTEST.COM (short username oryx)
I20260806 02:08:01.067867 4359 webserver.cc:460] Webserver started at http://127.28.21.1:42355/ using document root <none> and password file <none>
I20260806 02:08:01.068435 4359 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:01.068495 4359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:01.068739 4359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:01.070839 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "df355d001a184e6995f18b7ecd7432f6"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "6cc2ba236b41a330694ce51554d9e7e0"
server_key_iv: "747f4b5b20431fb1c5fb82dc65fc01df"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.071470 4359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "df355d001a184e6995f18b7ecd7432f6"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "6cc2ba236b41a330694ce51554d9e7e0"
server_key_iv: "747f4b5b20431fb1c5fb82dc65fc01df"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.075347 4359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:08:01.077736 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.079030 4359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:01.079164 4359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "df355d001a184e6995f18b7ecd7432f6"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "6cc2ba236b41a330694ce51554d9e7e0"
server_key_iv: "747f4b5b20431fb1c5fb82dc65fc01df"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.079274 4359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:01.091073 4359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:01.093986 4359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:01.094197 4359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:01.095613 4359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:01.095688 4359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.095763 4359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:01.095806 4359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:01.107143 4359 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:40909
I20260806 02:08:01.107223 4500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:40909 every 8 connection(s)
I20260806 02:08:01.108088 4359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:01.110471 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.096572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:38601 (local address 127.28.21.62:43607)
0806 02:08:01.096700 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.096704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.097699 (+ 995us) server_negotiation.cc:407] Connection header received
0806 02:08:01.098819 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.098823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.098885 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.099000 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.100098 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.100641 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.101866 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.102138 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.102255 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.106317 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.106321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.106324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.106358 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.108133 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.108771 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.108775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.108776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.108870 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.109247 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.109249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.109250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.109481 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.109562 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.110081 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.110222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:08:01.111096 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.097089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.1:38601)
0806 02:08:01.097551 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.097592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.098597 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.099170 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.099176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.099494 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.099948 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.099962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.100769 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.100778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.101737 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.101746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.102292 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.102297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.102551 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.103199 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.103225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.106140 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.108292 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.108298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.108319 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.108644 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.108977 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.108981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.108983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.109150 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.109576 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.109580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.109802 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.109959 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.110209 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260806 02:08:01.112577 4501 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43607
I20260806 02:08:01.112727 4501 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:01.113209 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 4359
I20260806 02:08:01.113336 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:08:01.113317 4501 heartbeater.cc:511] Master 127.28.21.62:43607 requested a full tablet report, sending...
I20260806 02:08:01.113698 28756 external_mini_cluster.cc:1428] Setting key 46e89009416b891a4366cf3f7ef3cdca
I20260806 02:08:01.114984 4282 ts_manager.cc:194] Registered new tserver with Master: df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909)
I20260806 02:08:01.116146 4282 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:38601
I20260806 02:08:01.124581 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.114803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:36695 (local address 127.28.21.62:43607)
0806 02:08:01.116748 (+ 1945us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.116754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.116768 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:01.116800 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.116802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.116845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.116955 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.117527 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.118262 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.119559 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.119771 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.119853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.121210 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.121213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.121215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.121237 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.122730 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.123265 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.123269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.123270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.123370 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.123694 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.123697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.123698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.123882 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.123969 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.124347 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.124460 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1875}
I20260806 02:08:01.124624 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.114757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.1:36695)
0806 02:08:01.115022 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.115038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.115116 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.116973 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.116976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.117242 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.117391 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.117401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.118386 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.118390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.119431 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.119445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.119918 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.119925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.120064 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.120575 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.120590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.121064 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.122838 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.122842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.122845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.123131 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.123482 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.123487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.123489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.123594 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.123994 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.123999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.124114 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.124245 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.124420 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
I20260806 02:08:01.146150 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43607
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:01.259572 4510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:01.259836 4510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:01.259883 4510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:01.264376 4510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:01.264434 4510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:01.264477 4510 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:08:01.264541 4510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:01.270517 4510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:01.271522 4510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:01.272517 4510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:01.282490 4516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:01.282606 4510 server_base.cc:1034] running on GCE node
W20260806 02:08:01.282497 4515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:01.282524 4518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:01.283258 4510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:01.283931 4510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:01.285122 4510 hybrid_clock.cc:648] HybridClock initialized: now 1785982081285106 us; error 27 us; skew 500 ppm
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:01.288022 4510 init.cc:377] Logged in from keytab as oryx/127.28.21.2@KRBTEST.COM (short username oryx)
I20260806 02:08:01.288883 4510 webserver.cc:460] Webserver started at http://127.28.21.2:46779/ using document root <none> and password file <none>
I20260806 02:08:01.289470 4510 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:01.289563 4510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:01.289798 4510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:01.291879 4510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "11e233df70424577b62eb293e60b3c9e"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "d8d5418f63a51ed76982f292f68fb1a3"
server_key_iv: "879c2ea3e1fce9b057a46ae475c39901"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.292441 4510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "11e233df70424577b62eb293e60b3c9e"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "d8d5418f63a51ed76982f292f68fb1a3"
server_key_iv: "879c2ea3e1fce9b057a46ae475c39901"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.296258 4510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:08:01.298635 4525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.300136 4510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:08:01.300259 4510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "11e233df70424577b62eb293e60b3c9e"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "d8d5418f63a51ed76982f292f68fb1a3"
server_key_iv: "879c2ea3e1fce9b057a46ae475c39901"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.300374 4510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:01.325966 4510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:01.329438 4510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:01.329598 4510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:01.331128 4510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:01.331204 4510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.331298 4510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:01.331324 4510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:01.342815 4510 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:40129
I20260806 02:08:01.343015 4646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:40129 every 8 connection(s)
I20260806 02:08:01.343845 4510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:01.345870 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.331833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:55703 (local address 127.28.21.62:43607)
0806 02:08:01.331994 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.331998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.333017 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 02:08:01.334119 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.334123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.334187 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.334249 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.335648 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.336292 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.337692 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.338078 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.338203 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.342188 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.342191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.342194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.342242 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.343926 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.344490 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.344494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.344496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.344590 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.344929 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.344932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.344933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.345109 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.345186 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.345652 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.345749 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:08:01.346735 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.332386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.2:55703)
0806 02:08:01.332856 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.332900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.333866 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.334701 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.334707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.335022 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.335496 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.335510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.336435 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.336438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.337558 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.337568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.338252 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.338256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.338504 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.339093 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.339120 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.341948 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.344033 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.344038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.344052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.344354 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.344679 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.344682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.344684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.344843 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.345206 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.345211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.345441 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.345598 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.345831 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260806 02:08:01.347999 4647 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43607
I20260806 02:08:01.348151 4647 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:01.348737 4647 heartbeater.cc:511] Master 127.28.21.62:43607 requested a full tablet report, sending...
I20260806 02:08:01.349920 4283 ts_manager.cc:194] Registered new tserver with Master: 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129)
I20260806 02:08:01.350742 4283 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:55703
I20260806 02:08:01.353556 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 4510
I20260806 02:08:01.353709 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:08:01.354084 28756 external_mini_cluster.cc:1428] Setting key f2ff6ba5498f34fd43a8d8b8dca59b89
I20260806 02:08:01.358620 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.350549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:33607 (local address 127.28.21.62:43607)
0806 02:08:01.350678 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.350682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.350892 (+ 210us) server_negotiation.cc:407] Connection header received
0806 02:08:01.351086 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.351089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.351135 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.351227 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.351752 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.352232 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.353350 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.353603 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.353724 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.355035 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.355039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.355042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.355068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.356542 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.357080 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.357083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.357084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.357156 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.357508 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.357521 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.357523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.357835 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.357971 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.358267 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.358426 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:08:01.358657 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.350461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.2:33607)
0806 02:08:01.350775 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.350788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.350943 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.351252 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.351254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.351503 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.351632 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.351640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.352375 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.352379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.353212 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.353222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.353726 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.353730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.353844 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.354357 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.354369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.354894 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.356659 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.356666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.356671 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.356982 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.357249 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.357255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.357261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.357375 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.357970 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.357973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.358078 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.358205 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.358459 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
I20260806 02:08:01.386845 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:43607
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:01.500564 4656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:01.500831 4656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:01.500875 4656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:01.504981 4656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:01.505039 4656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:01.505077 4656 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:08:01.505110 4656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:01.510176 4656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.21.62:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:01.511238 4656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:01.512270 4656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:08:01.522421 4656 server_base.cc:1034] running on GCE node
W20260806 02:08:01.522363 4661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:01.522451 4664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:01.522599 4662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:01.523052 4656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:01.523689 4656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:01.524856 4656 hybrid_clock.cc:648] HybridClock initialized: now 1785982081524835 us; error 31 us; skew 500 ppm
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:01.527750 4656 init.cc:377] Logged in from keytab as oryx/127.28.21.3@KRBTEST.COM (short username oryx)
I20260806 02:08:01.528647 4656 webserver.cc:460] Webserver started at http://127.28.21.3:40493/ using document root <none> and password file <none>
I20260806 02:08:01.529207 4656 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:01.529311 4656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:01.529552 4656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:01.531665 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "9b114606270d429d8614b4a666c709c2"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "b0d0161d9b84f0b2557b9f82bd40df53"
server_key_iv: "8d11ab80dabf6cb9e2cd9994acc147b1"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.532241 4656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "9b114606270d429d8614b4a666c709c2"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "b0d0161d9b84f0b2557b9f82bd40df53"
server_key_iv: "8d11ab80dabf6cb9e2cd9994acc147b1"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.535919 4656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:08:01.538271 4671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.539511 4656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:08:01.539654 4656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "9b114606270d429d8614b4a666c709c2"
format_stamp: "Formatted at 2026-08-06 02:08:01 on dist-test-slave-gp6n"
server_key: "b0d0161d9b84f0b2557b9f82bd40df53"
server_key_iv: "8d11ab80dabf6cb9e2cd9994acc147b1"
server_key_version: "encryptionkey@0"
I20260806 02:08:01.539765 4656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:01.569195 4656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:01.572726 4656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:01.572953 4656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:01.574512 4656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:01.574585 4656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:01.574662 4656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:01.574704 4656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:01.585980 4656 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:42651
I20260806 02:08:01.586074 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:42651 every 8 connection(s)
I20260806 02:08:01.587071 4656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:08:01.588857 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.575423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:45419 (local address 127.28.21.62:43607)
0806 02:08:01.575584 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.575589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.576486 (+ 897us) server_negotiation.cc:407] Connection header received
0806 02:08:01.577740 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.577745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.577809 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.577873 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.579598 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.580185 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.581364 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.581592 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.581709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.584961 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.584965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.584967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.585003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.586710 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.587285 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.587291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.587297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.587376 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.587770 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.587773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.587775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.588021 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.588124 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.588624 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.588735 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:08:01.589684 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.575841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.3:45419)
0806 02:08:01.576315 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.576365 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.577476 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.578322 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.578331 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.578801 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.579451 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.579465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.580319 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.580322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.581228 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.581237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.581760 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.581764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.581937 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.582594 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.582632 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.584776 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.586833 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.586839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.586858 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.587169 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.587472 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.587477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.587479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.587652 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.588129 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.588134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.588359 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.588572 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.588823 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":163,"threads_started":1}
I20260806 02:08:01.591061 4798 heartbeater.cc:347] Connected to a master server at 127.28.21.62:43607
I20260806 02:08:01.591233 4798 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:01.591765 4798 heartbeater.cc:511] Master 127.28.21.62:43607 requested a full tablet report, sending...
I20260806 02:08:01.592891 4283 ts_manager.cc:194] Registered new tserver with Master: 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651)
I20260806 02:08:01.593727 4283 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:45419
I20260806 02:08:01.594285 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 4656
I20260806 02:08:01.594480 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:08:01.594806 28756 external_mini_cluster.cc:1428] Setting key 9afa3c37b1aeda987f51b5a8976af579
I20260806 02:08:01.596995 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:08:01.601042 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.593226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:33693 (local address 127.28.21.62:43607)
0806 02:08:01.593340 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.593344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.593557 (+ 213us) server_negotiation.cc:407] Connection header received
0806 02:08:01.593695 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.593698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.593735 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.593800 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.594396 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.595155 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.596150 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.596406 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.596484 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.597629 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.597635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.597639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.597667 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.599353 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.599886 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.599890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.599891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.599968 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.600275 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.600277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.600278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.600454 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.600529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.600838 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.600932 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:08:01.601238 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.593170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.28.21.3:33693)
0806 02:08:01.593467 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.593482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.593577 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.593850 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.593853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.594096 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:01.594245 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.594253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.595289 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.595291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.596059 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.596068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.596517 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.596521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.596616 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.597163 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:01.597174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:01.597496 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:01.599489 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.599493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.599497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.599778 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.600056 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:01.600061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:01.600065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:01.600170 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:01.600556 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:01.600561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:01.600668 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.600790 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.600957 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:08:01.624090 4335 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:01.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.28.21.62:43607)
0806 02:08:01.617659 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.617662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.617677 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:08:01.617728 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.617731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.617773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.617910 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.618781 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.619298 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.620174 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.620398 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.620502 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.623883 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57246: BlockingRecv error: recv got EOF from 127.0.0.1:57246 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 02:08:01 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982081, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:01.634605 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.625621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.28.21.62:43607)
0806 02:08:01.625748 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.625751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.625877 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:08:01.625977 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.625979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.626016 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.626086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:01.626750 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.627253 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.628312 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.628542 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.628628 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.631267 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:01.631271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:01.631273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:01.631296 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:01.632798 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.633247 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.633250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.633252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.633339 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:01.633711 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:01.633713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:01.633715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:01.633909 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:01.634025 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.634252 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.634459 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:08:01.637684 4283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:08:01.640782 4283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:01.655444 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.650327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.0.0.1:60598)
0806 02:08:01.650653 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.650665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.650740 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.651526 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.651529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.651547 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:01.651652 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.651659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.652908 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.652911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.654009 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.654019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.655019 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.655023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.655109 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.655131 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.655205 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.655274 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":54,"thread_start_us":115,"threads_started":1}
I20260806 02:08:01.655442 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.650903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.28.21.1:40909)
0806 02:08:01.651151 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.651157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.651178 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:08:01.651248 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.651252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.651389 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.651542 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:01.651750 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.652755 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.654125 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.654916 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.655020 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.655150 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.655233 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.655326 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260806 02:08:01.655293 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.649261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42651 (local address 127.0.0.1:33302)
0806 02:08:01.649772 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.649836 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.649972 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.650770 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.650778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.650805 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:01.650937 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.650943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.652313 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.652317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.653657 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.653666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.654760 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.654763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.654880 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.654923 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.655030 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.655142 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":34,"thread_start_us":83,"threads_started":1}
I20260806 02:08:01.656240 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.650016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40129 (local address 127.0.0.1:32812)
0806 02:08:01.650592 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.650606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.650698 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.651928 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.651931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.651958 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:01.652045 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.652050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.653461 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.653465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.654501 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.654517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.655821 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.655826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.655929 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.655963 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.656040 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.656105 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260806 02:08:01.656440 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.650923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.28.21.2:40129)
0806 02:08:01.651388 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.651399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.651422 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:08:01.651492 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.651497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.651667 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.651805 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:01.652178 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.653338 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.654622 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.655459 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.655568 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.655739 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.656088 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.656178 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":127,"threads_started":1}
I20260806 02:08:01.656752 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.649446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.28.21.3:42651)
0806 02:08:01.649792 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.649803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.649929 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:08:01.650148 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.650154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.650371 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.650579 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:01.651080 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.652209 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.653795 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.654636 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.654752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.654919 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.655363 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.655482 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260806 02:08:01.658372 4426 tablet_service.cc:1467] Processing CreateTablet for tablet ace5cd29b9aa46428e95faa6072f1252 (DEFAULT_TABLE table=test-table [id=e18960e9864046c18549add55557c882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.659113 4726 tablet_service.cc:1467] Processing CreateTablet for tablet ace5cd29b9aa46428e95faa6072f1252 (DEFAULT_TABLE table=test-table [id=e18960e9864046c18549add55557c882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.660032 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace5cd29b9aa46428e95faa6072f1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.661123 4578 tablet_service.cc:1467] Processing CreateTablet for tablet ace5cd29b9aa46428e95faa6072f1252 (DEFAULT_TABLE table=test-table [id=e18960e9864046c18549add55557c882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.661904 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace5cd29b9aa46428e95faa6072f1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.659502 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace5cd29b9aa46428e95faa6072f1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.666960 4829 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Bootstrap starting.
I20260806 02:08:01.669270 4829 tablet_bootstrap.cc:654] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.669920 4830 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Bootstrap starting.
I20260806 02:08:01.670159 4829 log.cc:824] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:01.671823 4830 tablet_bootstrap.cc:654] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.672497 4830 log.cc:824] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:01.674288 4830 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: No bootstrap required, opened a new log
I20260806 02:08:01.674518 4830 ts_tablet_manager.cc:1397] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:08:01.675009 4833 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Bootstrap starting.
I20260806 02:08:01.676648 4829 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: No bootstrap required, opened a new log
I20260806 02:08:01.676939 4829 ts_tablet_manager.cc:1397] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260806 02:08:01.677112 4833 tablet_bootstrap.cc:654] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.677930 4833 log.cc:824] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:01.678745 4830 raft_consensus.cc:348] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.679072 4830 raft_consensus.cc:374] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.679136 4830 raft_consensus.cc:729] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.679637 4830 consensus_queue.cc:260] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.679879 4833 tablet_bootstrap.cc:492] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: No bootstrap required, opened a new log
I20260806 02:08:01.680064 4833 ts_tablet_manager.cc:1397] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:08:01.680388 4647 heartbeater.cc:503] Master 127.28.21.62:43607 was elected leader, sending a full tablet report...
I20260806 02:08:01.680791 4830 ts_tablet_manager.cc:1428] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:08:01.681248 4829 raft_consensus.cc:348] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.681557 4829 raft_consensus.cc:374] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.681607 4829 raft_consensus.cc:729] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.682063 4829 consensus_queue.cc:260] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.682821 4798 heartbeater.cc:503] Master 127.28.21.62:43607 was elected leader, sending a full tablet report...
I20260806 02:08:01.683182 4829 ts_tablet_manager.cc:1428] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260806 02:08:01.685303 4833 raft_consensus.cc:348] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.685585 4833 raft_consensus.cc:374] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.685647 4833 raft_consensus.cc:729] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.686111 4833 consensus_queue.cc:260] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.686784 4501 heartbeater.cc:503] Master 127.28.21.62:43607 was elected leader, sending a full tablet report...
I20260806 02:08:01.687135 4833 ts_tablet_manager.cc:1428] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 02:08:01.832737 4838 raft_consensus.cc:482] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:01.832958 4838 raft_consensus.cc:504] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.834148 4838 leader_election.cc:288] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129), df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909)
W20260806 02:08:01.838063 4799 tablet.cc:2369] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:08:01.838577 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.834555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:41613 (local address 127.28.21.2:40129)
0806 02:08:01.834751 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.834757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.835130 (+ 373us) server_negotiation.cc:407] Connection header received
0806 02:08:01.835196 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.835200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.835254 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.835326 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:01.835766 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.836346 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.837419 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.837838 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.837982 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.838052 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.838349 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.838452 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:08:01.838585 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.834493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40129 (local address 127.28.21.3:41613)
0806 02:08:01.834862 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.834878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.834987 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.835458 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.835462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.835484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:01.835627 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.835635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.836468 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.836474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.837275 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.837284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.837998 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.838002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.838214 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.838245 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.838319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.838429 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":74}
I20260806 02:08:01.839258 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace5cd29b9aa46428e95faa6072f1252" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e" is_pre_election: true
I20260806 02:08:01.839618 4600 raft_consensus.cc:2454] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:01.840086 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.835266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:45801 (local address 127.28.21.1:40909)
0806 02:08:01.835457 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.835461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.835640 (+ 179us) server_negotiation.cc:407] Connection header received
0806 02:08:01.835854 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.835860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.835921 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.835969 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:01.836665 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.837161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.838347 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.838984 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.839089 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.839165 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.839888 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.839971 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 02:08:01.840090 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.834779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.28.21.3:45801)
0806 02:08:01.835543 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:01.835561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:01.835721 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:01.836080 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:01.836082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:01.836095 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:01.836545 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.836550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.837283 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.837288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.838203 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:01.838212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.839591 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.839595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:01.839703 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.839726 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:08:01.839843 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:08:01.839926 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":54,"thread_start_us":153,"threads_started":1}
I20260806 02:08:01.840130 4675 leader_election.cc:302] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, 9b114606270d429d8614b4a666c709c2; no voters:
I20260806 02:08:01.840416 4838 raft_consensus.cc:2781] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:01.840508 4838 raft_consensus.cc:482] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:01.840561 4838 raft_consensus.cc:3037] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:01.840932 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace5cd29b9aa46428e95faa6072f1252" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6" is_pre_election: true
I20260806 02:08:01.841230 4451 raft_consensus.cc:2454] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:01.841954 4838 raft_consensus.cc:504] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.842319 4838 leader_election.cc:288] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Requested vote from peers 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129), df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909)
I20260806 02:08:01.842623 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace5cd29b9aa46428e95faa6072f1252" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e"
I20260806 02:08:01.842715 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace5cd29b9aa46428e95faa6072f1252" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6"
I20260806 02:08:01.842742 4600 raft_consensus.cc:3037] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:01.842854 4451 raft_consensus.cc:3037] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:01.843999 4451 raft_consensus.cc:2454] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:01.844061 4600 raft_consensus.cc:2454] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:01.844260 4675 leader_election.cc:302] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b114606270d429d8614b4a666c709c2, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:01.844461 4838 raft_consensus.cc:2781] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:01.844677 4838 raft_consensus.cc:686] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Running, Role: LEADER
W20260806 02:08:01.844988 4648 tablet.cc:2369] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:08:01.845052 4838 consensus_queue.cc:237] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } }
I20260806 02:08:01.848644 4282 catalog_manager.cc:5495] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b114606270d429d8614b4a666c709c2 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "9b114606270d429d8614b4a666c709c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } health_report { overall_health: UNKNOWN } } }
W20260806 02:08:01.859602 4502 tablet.cc:2369] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:08:01.889442 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:01.885736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.28.21.3:42651)
0806 02:08:01.885870 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:01.885873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:01.885886 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:08:01.885963 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:01.885965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:01.886020 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:01.886109 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:01.886317 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.886900 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.887894 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:01.888144 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:01.888244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:01.888542 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:01.889023 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:01.889122 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:01.889243 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:08:01.889312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:08:01.895628 4600 raft_consensus.cc:1261] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:01.895628 4451 raft_consensus.cc:1261] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:01.896324 4838 consensus_queue.cc:1035] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:01.896572 4841 consensus_queue.cc:1035] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:01.905351 4846 mvcc.cc:205] Tried to move back new op lower bound from 7315382607436730368 to 7315382607239933952. Current Snapshot: MvccSnapshot[applied={T|T < 7315382607436730368}]
I20260806 02:08:01.914552 4281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:01.914736 4281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:01.917248 4281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Sending DeleteTablet for 3 replicas of tablet ace5cd29b9aa46428e95faa6072f1252
I20260806 02:08:01.918102 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet ace5cd29b9aa46428e95faa6072f1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:01 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33302
I20260806 02:08:01.918464 4426 tablet_service.cc:1514] Processing DeleteTablet for tablet ace5cd29b9aa46428e95faa6072f1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:01 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:60598
I20260806 02:08:01.918644 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet ace5cd29b9aa46428e95faa6072f1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:01 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260806 02:08:01.918958 4281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:08:01.919149 4856 tablet_replica.cc:330] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: stopping tablet replica
I20260806 02:08:01.919267 4857 tablet_replica.cc:330] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: stopping tablet replica
I20260806 02:08:01.919462 4856 raft_consensus.cc:2229] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:08:01.919596 4857 raft_consensus.cc:2229] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 02:08:01.919735 4281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:01.919875 4856 raft_consensus.cc:2258] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:01.919950 4857 raft_consensus.cc:2258] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:01.921082 4858 tablet_replica.cc:330] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: stopping tablet replica
I20260806 02:08:01.921429 4858 raft_consensus.cc:2229] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:01.921497 4856 ts_tablet_manager.cc:1905] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:01.921804 4858 raft_consensus.cc:2258] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:01.923606 4857 ts_tablet_manager.cc:1905] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:01.925082 4856 ts_tablet_manager.cc:1918] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:01.925192 4856 log.cc:1196] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/ace5cd29b9aa46428e95faa6072f1252
I20260806 02:08:01.925535 4856 ts_tablet_manager.cc:1939] T ace5cd29b9aa46428e95faa6072f1252 P 9b114606270d429d8614b4a666c709c2: Deleting consensus metadata
I20260806 02:08:01.926590 4857 ts_tablet_manager.cc:1918] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:01.926700 4857 log.cc:1196] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/ace5cd29b9aa46428e95faa6072f1252
I20260806 02:08:01.926786 4270 catalog_manager.cc:4841] TS 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651): tablet ace5cd29b9aa46428e95faa6072f1252 (table test-table [id=e18960e9864046c18549add55557c882]) successfully deleted
I20260806 02:08:01.927066 4857 ts_tablet_manager.cc:1939] T ace5cd29b9aa46428e95faa6072f1252 P df355d001a184e6995f18b7ecd7432f6: Deleting consensus metadata
I20260806 02:08:01.927587 4426 tablet_service.cc:1467] Processing CreateTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (DEFAULT_TABLE table=test-table [id=2edb02c856044a508387dcc033963dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.927884 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.927917 4578 tablet_service.cc:1467] Processing CreateTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (DEFAULT_TABLE table=test-table [id=2edb02c856044a508387dcc033963dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.928200 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.928205 4858 ts_tablet_manager.cc:1905] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:01.930317 4833 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Bootstrap starting.
I20260806 02:08:01.930586 4858 ts_tablet_manager.cc:1918] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:01.930676 4858 log.cc:1196] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/ace5cd29b9aa46428e95faa6072f1252
I20260806 02:08:01.930791 4830 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Bootstrap starting.
I20260806 02:08:01.931160 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (DEFAULT_TABLE table=test-table [id=2edb02c856044a508387dcc033963dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:01.931434 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:01.931509 4833 tablet_bootstrap.cc:654] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.931835 4830 tablet_bootstrap.cc:654] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.933557 4830 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: No bootstrap required, opened a new log
I20260806 02:08:01.933670 4830 ts_tablet_manager.cc:1397] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:08:01.933766 4829 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Bootstrap starting.
I20260806 02:08:01.934106 4858 ts_tablet_manager.cc:1939] T ace5cd29b9aa46428e95faa6072f1252 P 11e233df70424577b62eb293e60b3c9e: Deleting consensus metadata
I20260806 02:08:01.934157 4830 raft_consensus.cc:348] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.934283 4830 raft_consensus.cc:374] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.934317 4830 raft_consensus.cc:729] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.934499 4830 consensus_queue.cc:260] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.934760 4829 tablet_bootstrap.cc:654] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:01.935799 4270 catalog_manager.cc:4841] TS 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129): tablet ace5cd29b9aa46428e95faa6072f1252 (table test-table [id=e18960e9864046c18549add55557c882]) successfully deleted
I20260806 02:08:01.936105 4833 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: No bootstrap required, opened a new log
I20260806 02:08:01.936187 4833 ts_tablet_manager.cc:1397] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 02:08:01.936529 4830 ts_tablet_manager.cc:1428] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260806 02:08:01.936954 4833 raft_consensus.cc:348] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.937080 4833 raft_consensus.cc:374] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.937122 4833 raft_consensus.cc:729] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.937254 4833 consensus_queue.cc:260] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.937538 4833 ts_tablet_manager.cc:1428] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 02:08:01.938211 4270 catalog_manager.cc:4841] TS df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909): tablet ace5cd29b9aa46428e95faa6072f1252 (table test-table [id=e18960e9864046c18549add55557c882]) successfully deleted
I20260806 02:08:01.939157 4829 tablet_bootstrap.cc:492] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: No bootstrap required, opened a new log
I20260806 02:08:01.939277 4829 ts_tablet_manager.cc:1397] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:08:01.939702 4829 raft_consensus.cc:348] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.939839 4829 raft_consensus.cc:374] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:01.939877 4829 raft_consensus.cc:729] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Initialized, Role: FOLLOWER
I20260806 02:08:01.940018 4829 consensus_queue.cc:260] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:01.940320 4829 ts_tablet_manager.cc:1428] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:08:02.030707 4835 raft_consensus.cc:482] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:02.030849 4835 raft_consensus.cc:504] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.031981 4835 leader_election.cc:288] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651)
I20260806 02:08:02.036640 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.032709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:37907 (local address 127.28.21.3:42651)
0806 02:08:02.032874 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.032878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.033097 (+ 219us) server_negotiation.cc:407] Connection header received
0806 02:08:02.033167 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.033170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.033220 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.033310 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:02.033599 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.034354 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.035690 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.036076 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.036195 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.036258 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.036457 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.036524 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:08:02.036657 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.032605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42651 (local address 127.28.21.2:37907)
0806 02:08:02.032991 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.033006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.033093 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.033342 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.033348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.033371 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:02.033474 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.033480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.034510 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.034513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.035575 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.035583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.036206 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.036209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.036320 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.036339 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.036430 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.036500 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":112,"threads_started":1}
I20260806 02:08:02.036830 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.032321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:46701 (local address 127.28.21.1:40909)
0806 02:08:02.032480 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.032485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.032575 (+ 90us) server_negotiation.cc:407] Connection header received
0806 02:08:02.032806 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.032810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.032870 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.032923 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:02.033386 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.033994 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.035456 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.036075 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.036187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.036255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.036612 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.036695 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:08:02.037163 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764b7c76a1f4509b1f0bfda30fb3eb9" candidate_uuid: "11e233df70424577b62eb293e60b3c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b114606270d429d8614b4a666c709c2" is_pre_election: true
I20260806 02:08:02.037271 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.032196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.28.21.2:46701)
0806 02:08:02.032479 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.032497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.032661 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.033083 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.033086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.033098 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:02.033239 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.033244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.034110 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.034112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.035308 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.035320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.036330 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.036335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.036451 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.036479 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.037048 (+ 569us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.037130 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":113,"threads_started":1}
I20260806 02:08:02.037390 4748 raft_consensus.cc:2454] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11e233df70424577b62eb293e60b3c9e in term 0.
I20260806 02:08:02.037731 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764b7c76a1f4509b1f0bfda30fb3eb9" candidate_uuid: "11e233df70424577b62eb293e60b3c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6" is_pre_election: true
I20260806 02:08:02.037885 4451 raft_consensus.cc:2454] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11e233df70424577b62eb293e60b3c9e in term 0.
I20260806 02:08:02.037871 4529 leader_election.cc:302] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, 9b114606270d429d8614b4a666c709c2; no voters:
I20260806 02:08:02.038151 4835 raft_consensus.cc:2781] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:02.038220 4835 raft_consensus.cc:482] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:02.038261 4835 raft_consensus.cc:3037] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.039299 4835 raft_consensus.cc:504] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.039664 4835 leader_election.cc:288] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [CANDIDATE]: Term 1 election: Requested vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651)
I20260806 02:08:02.040045 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764b7c76a1f4509b1f0bfda30fb3eb9" candidate_uuid: "11e233df70424577b62eb293e60b3c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6"
I20260806 02:08:02.040083 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764b7c76a1f4509b1f0bfda30fb3eb9" candidate_uuid: "11e233df70424577b62eb293e60b3c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b114606270d429d8614b4a666c709c2"
I20260806 02:08:02.040167 4451 raft_consensus.cc:3037] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.040181 4748 raft_consensus.cc:3037] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.041344 4451 raft_consensus.cc:2454] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11e233df70424577b62eb293e60b3c9e in term 1.
I20260806 02:08:02.041344 4748 raft_consensus.cc:2454] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11e233df70424577b62eb293e60b3c9e in term 1.
I20260806 02:08:02.041620 4529 leader_election.cc:302] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:02.041777 4835 raft_consensus.cc:2781] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:02.041980 4835 raft_consensus.cc:686] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 1 LEADER]: Becoming Leader. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Running, Role: LEADER
I20260806 02:08:02.042227 4835 consensus_queue.cc:237] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.044736 4282 catalog_manager.cc:5495] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 11e233df70424577b62eb293e60b3c9e (127.28.21.2). New cstate: current_term: 1 leader_uuid: "11e233df70424577b62eb293e60b3c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:08:02 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982080, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.62@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:02.100394 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.092470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:43607 (local address 127.0.0.1:57264)
0806 02:08:02.092779 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.092792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.092899 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.093122 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.093124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.093522 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:02.093618 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.093624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.094321 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.094323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.094891 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.094898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.095292 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.095294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.095353 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.095909 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:02.095920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:02.097314 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:02.099010 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:02.099018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:02.099022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:02.099242 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:02.099499 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:02.099501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:02.099502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:02.099580 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:02.099901 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:02.099906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:02.100004 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.100092 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.100211 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":128,"threads_started":1}
I20260806 02:08:02.100396 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.092524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.28.21.62:43607)
0806 02:08:02.092665 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.092669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.092866 (+ 197us) server_negotiation.cc:407] Connection header received
0806 02:08:02.092969 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.092972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.093012 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.093081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:02.093723 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.094231 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.094990 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.095193 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.095269 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.097439 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:02.097462 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:02.097464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:02.097485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:02.098923 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:02.099326 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:02.099328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:02.099329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:02.099400 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:02.099656 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:02.099658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:02.099659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:02.099817 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:02.099890 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.100147 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.100266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":55}
I20260806 02:08:02.104656 4282 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:57264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:08:02.105427 4282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:02.110296 4426 tablet_service.cc:1467] Processing CreateTablet for tablet 39f16236ea5c4beba4780181f86df011 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fdfa70fa724491a83569aaf5e9b3d93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:08:02.110507 4578 tablet_service.cc:1467] Processing CreateTablet for tablet 39f16236ea5c4beba4780181f86df011 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fdfa70fa724491a83569aaf5e9b3d93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:08:02.110706 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f16236ea5c4beba4780181f86df011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.110885 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f16236ea5c4beba4780181f86df011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.110886 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 39f16236ea5c4beba4780181f86df011 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fdfa70fa724491a83569aaf5e9b3d93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:08:02.111244 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f16236ea5c4beba4780181f86df011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.113652 4833 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: Bootstrap starting.
I20260806 02:08:02.113889 4829 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: Bootstrap starting.
I20260806 02:08:02.114653 4833 tablet_bootstrap.cc:654] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.114715 4829 tablet_bootstrap.cc:654] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.115661 4830 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: Bootstrap starting.
I20260806 02:08:02.115759 4833 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: No bootstrap required, opened a new log
I20260806 02:08:02.115833 4833 ts_tablet_manager.cc:1397] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:02.116187 4833 raft_consensus.cc:348] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.116284 4833 raft_consensus.cc:374] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.116314 4833 raft_consensus.cc:729] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.116434 4833 consensus_queue.cc:260] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.116676 4833 ts_tablet_manager.cc:1428] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:08:02.116700 4830 tablet_bootstrap.cc:654] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.116684 4839 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.116860 4839 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.117391 4829 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: No bootstrap required, opened a new log
I20260806 02:08:02.117486 4829 ts_tablet_manager.cc:1397] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:08:02.117861 4829 raft_consensus.cc:348] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.117950 4829 raft_consensus.cc:374] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.117978 4829 raft_consensus.cc:729] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.118084 4829 consensus_queue.cc:260] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.118311 4829 ts_tablet_manager.cc:1428] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:08:02.118381 4830 tablet_bootstrap.cc:492] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: No bootstrap required, opened a new log
I20260806 02:08:02.118494 4830 ts_tablet_manager.cc:1397] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260806 02:08:02.118857 4830 raft_consensus.cc:348] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.118986 4830 raft_consensus.cc:374] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.119024 4830 raft_consensus.cc:729] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.119149 4830 consensus_queue.cc:260] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.119426 4830 ts_tablet_manager.cc:1428] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:08:02.119369 4835 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.119498 4835 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.118319 4854 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.120412 4854 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.530645 4835 consensus_queue.cc:1035] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.535037 4864 consensus_queue.cc:1035] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.572616 4839 raft_consensus.cc:482] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:02.572759 4839 raft_consensus.cc:504] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.573976 4839 leader_election.cc:288] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129), 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651)
I20260806 02:08:02.578377 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.574332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33699 (local address 127.28.21.2:40129)
0806 02:08:02.574616 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.574621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.574712 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:08:02.574906 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.574912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.574988 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.575076 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:02.575437 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.576044 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577301 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.577713 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577796 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.577849 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.578116 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.578210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260806 02:08:02.578540 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.574213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40129 (local address 127.28.21.1:33699)
0806 02:08:02.574575 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.574613 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.574763 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.575116 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.575120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.575145 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:02.575302 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.575312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.576160 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.576163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.577172 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.577182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.577840 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.577977 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.578002 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.578085 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.578182 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260806 02:08:02.578577 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.574800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:35479 (local address 127.28.21.3:42651)
0806 02:08:02.575093 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.575098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.575113 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:08:02.575167 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.575170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.575218 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.575317 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:02.575619 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.576214 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577305 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.577802 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577954 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.578026 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.578205 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.578440 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260806 02:08:02.578824 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.574655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42651 (local address 127.28.21.1:35479)
0806 02:08:02.574903 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.574916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.575001 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.575338 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.575340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.575355 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:02.575499 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.575506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.576330 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.576333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.577173 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.577183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.577956 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.577959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.578046 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.578094 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.578172 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.578238 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260806 02:08:02.579048 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f16236ea5c4beba4780181f86df011" candidate_uuid: "df355d001a184e6995f18b7ecd7432f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e" is_pre_election: true
I20260806 02:08:02.579226 4600 raft_consensus.cc:2454] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df355d001a184e6995f18b7ecd7432f6 in term 0.
I20260806 02:08:02.579265 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f16236ea5c4beba4780181f86df011" candidate_uuid: "df355d001a184e6995f18b7ecd7432f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b114606270d429d8614b4a666c709c2" is_pre_election: true
I20260806 02:08:02.579448 4748 raft_consensus.cc:2454] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df355d001a184e6995f18b7ecd7432f6 in term 0.
I20260806 02:08:02.579608 4378 leader_election.cc:302] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:02.579880 4839 raft_consensus.cc:2781] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:02.579957 4839 raft_consensus.cc:482] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:02.580010 4839 raft_consensus.cc:3037] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.581017 4839 raft_consensus.cc:504] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.581359 4839 leader_election.cc:288] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [CANDIDATE]: Term 1 election: Requested vote from peers 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129), 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651)
I20260806 02:08:02.581748 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f16236ea5c4beba4780181f86df011" candidate_uuid: "df355d001a184e6995f18b7ecd7432f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e"
I20260806 02:08:02.581828 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f16236ea5c4beba4780181f86df011" candidate_uuid: "df355d001a184e6995f18b7ecd7432f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b114606270d429d8614b4a666c709c2"
I20260806 02:08:02.581885 4600 raft_consensus.cc:3037] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.581960 4748 raft_consensus.cc:3037] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.583120 4600 raft_consensus.cc:2454] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df355d001a184e6995f18b7ecd7432f6 in term 1.
I20260806 02:08:02.583120 4748 raft_consensus.cc:2454] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df355d001a184e6995f18b7ecd7432f6 in term 1.
I20260806 02:08:02.583400 4378 leader_election.cc:302] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:02.583602 4839 raft_consensus.cc:2781] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:02.583807 4839 raft_consensus.cc:686] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [term 1 LEADER]: Becoming Leader. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Running, Role: LEADER
I20260806 02:08:02.584110 4839 consensus_queue.cc:237] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.585413 4885 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: TxnStatusTablet state changed. Reason: New leader df355d001a184e6995f18b7ecd7432f6. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.585536 4885 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:08:02.585976 4887 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:08:02.586125 4887 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:08:02.586784 4281 catalog_manager.cc:5495] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to df355d001a184e6995f18b7ecd7432f6 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } health_report { overall_health: UNKNOWN } } }
I20260806 02:08:02.688805 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.684043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.0.0.1:60602)
0806 02:08:02.684315 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.684332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.684446 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.684815 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.684818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.684825 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:08:02.684920 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.684925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.686025 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.686028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.686857 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.686869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.687497 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.687500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.687574 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.687622 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.688513 (+ 891us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.688530 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.688600 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.688677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260806 02:08:02.688863 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.684134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.28.21.1:40909)
0806 02:08:02.684400 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.684407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.684425 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:08:02.684594 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.684597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.684670 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.684788 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:02.685032 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.685878 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.686995 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.687371 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.687485 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.687804 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.688383 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.688502 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.688637 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.688701 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":64,"threads_started":1}
I20260806 02:08:02.691903 4600 raft_consensus.cc:1261] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Refusing update from remote peer df355d001a184e6995f18b7ecd7432f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:02.691906 4748 raft_consensus.cc:1261] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Refusing update from remote peer df355d001a184e6995f18b7ecd7432f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:02.692471 4885 consensus_queue.cc:1035] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.692672 4839 consensus_queue.cc:1035] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.694242 4864 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: TxnStatusTablet state changed. Reason: New leader df355d001a184e6995f18b7ecd7432f6. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.694362 4864 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.694705 4854 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: TxnStatusTablet state changed. Reason: New leader df355d001a184e6995f18b7ecd7432f6. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.694834 4854 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.696589 4839 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.696705 4839 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:08:02.698975 4854 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.699100 4854 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 9b114606270d429d8614b4a666c709c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.700956 4839 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.701073 4839 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P df355d001a184e6995f18b7ecd7432f6: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:08:02.701998 4864 tablet_replica.cc:439] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df355d001a184e6995f18b7ecd7432f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } } }
I20260806 02:08:02.702121 4864 tablet_replica.cc:442] T 39f16236ea5c4beba4780181f86df011 P 11e233df70424577b62eb293e60b3c9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:08:02.710129 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.706335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.28.21.2:40129)
0806 02:08:02.706504 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.706508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.706642 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:08:02.706851 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.706854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.706915 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.707024 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:02.707272 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.707800 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.708677 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.708919 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.709025 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.709333 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.709732 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.709871 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.709932 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.709990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 02:08:02.721163 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.717654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:55683 (local address 127.28.21.1:40909)
0806 02:08:02.717788 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.717792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.717927 (+ 135us) server_negotiation.cc:407] Connection header received
0806 02:08:02.718022 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.718025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.718081 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.718168 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:02.718397 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.719215 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.720050 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.720367 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.720473 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.720716 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.720791 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.720862 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.720953 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.721031 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:08:02.721204 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.717502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.28.21.2:55683)
0806 02:08:02.717824 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.717846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.717937 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.718188 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.718190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.718195 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:08:02.718292 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.718296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.719321 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.719323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.719929 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.719937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.720485 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.720487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.720556 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.720595 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.720888 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.720893 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.720944 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.721029 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":148,"threads_started":1}
I20260806 02:08:02.746652 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.742571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40129 (local address 127.28.21.1:60251)
0806 02:08:02.742901 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:02.742922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:02.743006 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:02.743215 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:02.743217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:02.743223 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:08:02.743415 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.743420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.744475 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.744478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.745318 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:02.745328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.745892 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.745894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:02.745954 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.745995 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.746362 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.746367 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:08:02.746431 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:08:02.746536 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":129,"threads_started":1}
I20260806 02:08:02.746655 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.742616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:60251 (local address 127.28.21.2:40129)
0806 02:08:02.742767 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.742771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.742985 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:08:02.743075 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.743078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.743124 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.743199 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:02.743514 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.744372 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.745431 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.745788 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.745890 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.746138 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.746235 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.746314 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.746461 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.746518 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:08:02.772415 4282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:02.772563 4282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:02.774711 4282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Sending DeleteTablet for 3 replicas of tablet 4764b7c76a1f4509b1f0bfda30fb3eb9
I20260806 02:08:02.775254 4426 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:60598
I20260806 02:08:02.775300 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260806 02:08:02.775422 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33302
I20260806 02:08:02.775656 4909 tablet_replica.cc:330] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: stopping tablet replica
I20260806 02:08:02.775938 4910 tablet_replica.cc:330] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: stopping tablet replica
I20260806 02:08:02.775935 4909 raft_consensus.cc:2229] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:02.776049 4910 raft_consensus.cc:2229] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:08:02.776078 4909 raft_consensus.cc:2258] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.776345 4910 raft_consensus.cc:2258] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.776621 4911 tablet_replica.cc:330] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: stopping tablet replica
I20260806 02:08:02.776744 4911 raft_consensus.cc:2229] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:02.776851 4911 raft_consensus.cc:2258] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.776950 4909 ts_tablet_manager.cc:1905] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.777539 4911 ts_tablet_manager.cc:1905] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.780252 4909 ts_tablet_manager.cc:1918] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:08:02.780331 4909 log.cc:1196] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/4764b7c76a1f4509b1f0bfda30fb3eb9
I20260806 02:08:02.780615 4909 ts_tablet_manager.cc:1939] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P df355d001a184e6995f18b7ecd7432f6: Deleting consensus metadata
I20260806 02:08:02.780658 4911 ts_tablet_manager.cc:1918] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:08:02.780742 4911 log.cc:1196] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/4764b7c76a1f4509b1f0bfda30fb3eb9
I20260806 02:08:02.781020 4911 ts_tablet_manager.cc:1939] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 9b114606270d429d8614b4a666c709c2: Deleting consensus metadata
I20260806 02:08:02.781242 4910 ts_tablet_manager.cc:1905] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.781807 4270 catalog_manager.cc:4841] TS 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651): tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (table test-table [id=2edb02c856044a508387dcc033963dbe]) successfully deleted
I20260806 02:08:02.782167 4270 catalog_manager.cc:4841] TS df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909): tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (table test-table [id=2edb02c856044a508387dcc033963dbe]) successfully deleted
I20260806 02:08:02.784000 4910 ts_tablet_manager.cc:1918] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:08:02.784096 4910 log.cc:1196] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/4764b7c76a1f4509b1f0bfda30fb3eb9
I20260806 02:08:02.784379 4910 ts_tablet_manager.cc:1939] T 4764b7c76a1f4509b1f0bfda30fb3eb9 P 11e233df70424577b62eb293e60b3c9e: Deleting consensus metadata
I20260806 02:08:02.785009 4270 catalog_manager.cc:4841] TS 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129): tablet 4764b7c76a1f4509b1f0bfda30fb3eb9 (table test-table [id=2edb02c856044a508387dcc033963dbe]) successfully deleted
Aug 06 02:08:02 dist-test-slave-gp6n krb5kdc[4235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:08:02 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:02 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:08:02.807045 4919 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:02.800239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.28.21.62:43607)
0806 02:08:02.800495 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.800500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.800623 (+ 123us) server_negotiation.cc:407] Connection header received
0806 02:08:02.800805 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.800810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.800860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.800954 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:02.801678 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.802233 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.803188 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.803423 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.803533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.806863 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57272: BlockingRecv error: recv got EOF from 127.0.0.1:57272 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
Aug 06 02:08:02 dist-test-slave-gp6n krb5kdc[4235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM
I20260806 02:08:02.817226 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.808619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.28.21.62:43607)
0806 02:08:02.808724 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.808730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.808810 (+ 80us) server_negotiation.cc:407] Connection header received
0806 02:08:02.808926 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.808929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.808977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.809061 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:02.809662 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.810167 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.811025 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.811248 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.811354 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.813961 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:02.813964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:02.813966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:02.813990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:02.815547 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:02.816006 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:02.816009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:02.816010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:02.816087 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:02.816353 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:02.816355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:02.816356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:02.816542 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:02.816644 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.816890 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.817036 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:08:02.819861 4282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:08:02.820322 4282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:02.825198 4426 tablet_service.cc:1467] Processing CreateTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef (DEFAULT_TABLE table=test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.825266 4578 tablet_service.cc:1467] Processing CreateTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef (DEFAULT_TABLE table=test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.825333 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef (DEFAULT_TABLE table=test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.825531 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b5a6ab29c2494194d3aaacca1adeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.825531 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b5a6ab29c2494194d3aaacca1adeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.825630 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b5a6ab29c2494194d3aaacca1adeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.828509 4925 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Bootstrap starting.
I20260806 02:08:02.829457 4925 tablet_bootstrap.cc:654] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.830054 4927 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Bootstrap starting.
I20260806 02:08:02.830646 4925 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: No bootstrap required, opened a new log
I20260806 02:08:02.830684 4926 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Bootstrap starting.
I20260806 02:08:02.830752 4925 ts_tablet_manager.cc:1397] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:02.831190 4925 raft_consensus.cc:348] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.831342 4925 raft_consensus.cc:374] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.831391 4925 raft_consensus.cc:729] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.831553 4925 consensus_queue.cc:260] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.832062 4926 tablet_bootstrap.cc:654] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.832484 4925 ts_tablet_manager.cc:1428] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:08:02.833244 4927 tablet_bootstrap.cc:654] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.834929 4927 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: No bootstrap required, opened a new log
I20260806 02:08:02.835011 4927 ts_tablet_manager.cc:1397] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 02:08:02.835246 4926 tablet_bootstrap.cc:492] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: No bootstrap required, opened a new log
I20260806 02:08:02.835326 4926 ts_tablet_manager.cc:1397] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:08:02.835445 4927 raft_consensus.cc:348] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.835546 4927 raft_consensus.cc:374] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.835567 4927 raft_consensus.cc:729] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.835704 4927 consensus_queue.cc:260] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.835768 4926 raft_consensus.cc:348] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.835873 4926 raft_consensus.cc:374] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.835903 4926 raft_consensus.cc:729] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.835966 4927 ts_tablet_manager.cc:1428] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:08:02.836031 4926 consensus_queue.cc:260] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.836272 4926 ts_tablet_manager.cc:1428] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:08:02.939185 4854 raft_consensus.cc:482] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:02.939343 4854 raft_consensus.cc:504] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.939828 4854 leader_election.cc:288] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129)
I20260806 02:08:02.940409 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b5a6ab29c2494194d3aaacca1adeef" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6" is_pre_election: true
I20260806 02:08:02.940415 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b5a6ab29c2494194d3aaacca1adeef" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e" is_pre_election: true
I20260806 02:08:02.940586 4451 raft_consensus.cc:2454] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:02.940596 4600 raft_consensus.cc:2454] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:02.940958 4675 leader_election.cc:302] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e233df70424577b62eb293e60b3c9e, 9b114606270d429d8614b4a666c709c2; no voters:
I20260806 02:08:02.941174 4854 raft_consensus.cc:2781] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:02.941248 4854 raft_consensus.cc:482] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:02.941303 4854 raft_consensus.cc:3037] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.942381 4854 raft_consensus.cc:504] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.942770 4854 leader_election.cc:288] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Requested vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129)
I20260806 02:08:02.943131 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b5a6ab29c2494194d3aaacca1adeef" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6"
I20260806 02:08:02.943231 4451 raft_consensus.cc:3037] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.943233 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b5a6ab29c2494194d3aaacca1adeef" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e"
I20260806 02:08:02.943364 4600 raft_consensus.cc:3037] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:02.944329 4451 raft_consensus.cc:2454] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:02.944481 4600 raft_consensus.cc:2454] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:02.944659 4675 leader_election.cc:302] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b114606270d429d8614b4a666c709c2, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:02.944859 4854 raft_consensus.cc:2781] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:02.945096 4854 raft_consensus.cc:686] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Running, Role: LEADER
I20260806 02:08:02.945276 4854 consensus_queue.cc:237] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.947023 4281 catalog_manager.cc:5495] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b114606270d429d8614b4a666c709c2 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "9b114606270d429d8614b4a666c709c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } health_report { overall_health: HEALTHY } } }
I20260806 02:08:02.958089 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:02.955013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.28.21.3:42651)
0806 02:08:02.955146 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:02.955150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:02.955163 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:08:02.955276 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:02.955278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:02.955324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:02.955403 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:02.955675 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.956288 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.957170 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:02.957413 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:02.957500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:02.957727 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:02.957792 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:02.957855 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:02.957936 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 02:08:02.957982 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:08:02.960062 4600 raft_consensus.cc:1261] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:02.960090 4451 raft_consensus.cc:1261] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:02.960552 4854 consensus_queue.cc:1035] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.960740 4931 consensus_queue.cc:1035] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:02.969128 4282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:02.969301 4282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:02.971352 4282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Sending DeleteTablet for 3 replicas of tablet 61b5a6ab29c2494194d3aaacca1adeef
I20260806 02:08:02.971896 4426 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:60598
I20260806 02:08:02.972059 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260806 02:08:02.972203 4909 tablet_replica.cc:330] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: stopping tablet replica
I20260806 02:08:02.972256 4910 tablet_replica.cc:330] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: stopping tablet replica
I20260806 02:08:02.972337 4909 raft_consensus.cc:2229] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:02.972366 4910 raft_consensus.cc:2229] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:02.972461 4909 raft_consensus.cc:2258] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.972487 4910 raft_consensus.cc:2258] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.972640 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b5a6ab29c2494194d3aaacca1adeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:02 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33302
I20260806 02:08:02.972632 4282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:08:02.972916 4911 tablet_replica.cc:330] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: stopping tablet replica
W20260806 02:08:02.973061 4282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:02.973069 4911 raft_consensus.cc:2229] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:08:02.973260 4911 raft_consensus.cc:2258] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:02.973322 4909 ts_tablet_manager.cc:1905] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.973875 4910 ts_tablet_manager.cc:1905] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.976004 4909 ts_tablet_manager.cc:1918] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:02.976094 4909 log.cc:1196] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/61b5a6ab29c2494194d3aaacca1adeef
I20260806 02:08:02.976382 4909 ts_tablet_manager.cc:1939] T 61b5a6ab29c2494194d3aaacca1adeef P df355d001a184e6995f18b7ecd7432f6: Deleting consensus metadata
I20260806 02:08:02.976426 4910 ts_tablet_manager.cc:1918] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:02.976519 4910 log.cc:1196] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/61b5a6ab29c2494194d3aaacca1adeef
I20260806 02:08:02.976804 4910 ts_tablet_manager.cc:1939] T 61b5a6ab29c2494194d3aaacca1adeef P 11e233df70424577b62eb293e60b3c9e: Deleting consensus metadata
I20260806 02:08:02.977286 4270 catalog_manager.cc:4841] TS df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909): tablet 61b5a6ab29c2494194d3aaacca1adeef (table test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]) successfully deleted
I20260806 02:08:02.977895 4911 ts_tablet_manager.cc:1905] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:02.978130 4578 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 (DEFAULT_TABLE table=test-table [id=9b71a9860b5d4ced8fa95d9483eea083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.978371 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.978622 4426 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 (DEFAULT_TABLE table=test-table [id=9b71a9860b5d4ced8fa95d9483eea083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.978818 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.979388 4270 catalog_manager.cc:4841] TS 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129): tablet 61b5a6ab29c2494194d3aaacca1adeef (table test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]) successfully deleted
I20260806 02:08:02.979534 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 (DEFAULT_TABLE table=test-table [id=9b71a9860b5d4ced8fa95d9483eea083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:02.979769 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:02.980517 4911 ts_tablet_manager.cc:1918] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:02.980585 4911 log.cc:1196] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/61b5a6ab29c2494194d3aaacca1adeef
I20260806 02:08:02.980844 4911 ts_tablet_manager.cc:1939] T 61b5a6ab29c2494194d3aaacca1adeef P 9b114606270d429d8614b4a666c709c2: Deleting consensus metadata
I20260806 02:08:02.981278 4926 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Bootstrap starting.
I20260806 02:08:02.981859 4925 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Bootstrap starting.
I20260806 02:08:02.982106 4926 tablet_bootstrap.cc:654] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.982739 4925 tablet_bootstrap.cc:654] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.982822 4270 catalog_manager.cc:4841] TS 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651): tablet 61b5a6ab29c2494194d3aaacca1adeef (table test-table [id=30b9b85ae2ec49e5a4b893d1405225d8]) successfully deleted
I20260806 02:08:02.982903 4927 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: Bootstrap starting.
I20260806 02:08:02.983594 4927 tablet_bootstrap.cc:654] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:02.984331 4926 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: No bootstrap required, opened a new log
I20260806 02:08:02.984436 4926 ts_tablet_manager.cc:1397] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:08:02.984722 4927 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: No bootstrap required, opened a new log
I20260806 02:08:02.984829 4927 ts_tablet_manager.cc:1397] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:08:02.984822 4926 raft_consensus.cc:348] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.984963 4926 raft_consensus.cc:374] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.985033 4926 raft_consensus.cc:729] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e233df70424577b62eb293e60b3c9e, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.985168 4926 consensus_queue.cc:260] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.985210 4927 raft_consensus.cc:348] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.985342 4927 raft_consensus.cc:374] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.985380 4927 raft_consensus.cc:729] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df355d001a184e6995f18b7ecd7432f6, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.985433 4926 ts_tablet_manager.cc:1428] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:08:02.985510 4927 consensus_queue.cc:260] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.985788 4927 ts_tablet_manager.cc:1428] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:08:02.986636 4925 tablet_bootstrap.cc:492] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: No bootstrap required, opened a new log
I20260806 02:08:02.986755 4925 ts_tablet_manager.cc:1397] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:08:02.987112 4925 raft_consensus.cc:348] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.987229 4925 raft_consensus.cc:374] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:02.987270 4925 raft_consensus.cc:729] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Initialized, Role: FOLLOWER
I20260806 02:08:02.987403 4925 consensus_queue.cc:260] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:02.987679 4925 ts_tablet_manager.cc:1428] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:08:03.008699 4936 raft_consensus.cc:482] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:03.008857 4936 raft_consensus.cc:504] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:03.009272 4936 leader_election.cc:288] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129)
I20260806 02:08:03.009752 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee6e741cbe04a6f9c3fa8afd2c201d3" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6" is_pre_election: true
I20260806 02:08:03.009797 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee6e741cbe04a6f9c3fa8afd2c201d3" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e" is_pre_election: true
I20260806 02:08:03.009945 4451 raft_consensus.cc:2454] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:03.009953 4600 raft_consensus.cc:2454] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 0.
I20260806 02:08:03.010269 4675 leader_election.cc:302] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b114606270d429d8614b4a666c709c2, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:03.010495 4936 raft_consensus.cc:2781] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:03.010571 4936 raft_consensus.cc:482] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:03.010600 4936 raft_consensus.cc:3037] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:03.011777 4936 raft_consensus.cc:504] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:03.012148 4936 leader_election.cc:288] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Requested vote from peers df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909), 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129)
I20260806 02:08:03.012535 4451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee6e741cbe04a6f9c3fa8afd2c201d3" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df355d001a184e6995f18b7ecd7432f6"
I20260806 02:08:03.012667 4451 raft_consensus.cc:3037] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:03.012718 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee6e741cbe04a6f9c3fa8afd2c201d3" candidate_uuid: "9b114606270d429d8614b4a666c709c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e233df70424577b62eb293e60b3c9e"
I20260806 02:08:03.012890 4600 raft_consensus.cc:3037] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:03.013794 4451 raft_consensus.cc:2454] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:03.013819 4600 raft_consensus.cc:2454] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b114606270d429d8614b4a666c709c2 in term 1.
I20260806 02:08:03.014074 4675 leader_election.cc:302] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b114606270d429d8614b4a666c709c2, df355d001a184e6995f18b7ecd7432f6; no voters:
I20260806 02:08:03.014294 4936 raft_consensus.cc:2781] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:03.014444 4936 raft_consensus.cc:686] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9b114606270d429d8614b4a666c709c2, State: Running, Role: LEADER
I20260806 02:08:03.014609 4936 consensus_queue.cc:237] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } }
I20260806 02:08:03.016283 4281 catalog_manager.cc:5495] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b114606270d429d8614b4a666c709c2 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "9b114606270d429d8614b4a666c709c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b114606270d429d8614b4a666c709c2" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42651 } health_report { overall_health: HEALTHY } } }
I20260806 02:08:03.037011 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:03.033638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:50387 (local address 127.28.21.1:40909)
0806 02:08:03.033798 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:03.033803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:03.033970 (+ 167us) server_negotiation.cc:407] Connection header received
0806 02:08:03.034151 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:03.034154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:03.034198 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:03.034275 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:03.034565 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.035076 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.035940 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.036169 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.036278 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:03.036571 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:03.036640 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:03.036718 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:03.036805 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:08:03.036858 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:08:03.037011 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:03.033495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.28.21.3:50387)
0806 02:08:03.033881 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:03.033894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:03.034030 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:03.034298 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:03.034301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:03.034309 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:08:03.034441 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:03.034447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.035184 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.035189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:03.035817 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:03.035825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.036279 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.036282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:03.036372 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:03.036424 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:08:03.036721 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:08:03.036731 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:08:03.036797 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:08:03.036880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
I20260806 02:08:03.043159 4600 raft_consensus.cc:1261] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:03.043221 4451 raft_consensus.cc:1261] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Refusing update from remote peer 9b114606270d429d8614b4a666c709c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:03.043565 4936 consensus_queue.cc:1035] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e233df70424577b62eb293e60b3c9e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:03.043740 4854 consensus_queue.cc:1035] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df355d001a184e6995f18b7ecd7432f6" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:03.046546 4880 mvcc.cc:205] Tried to move back new op lower bound from 7315382612141260800 to 7315382612029169664. Current Snapshot: MvccSnapshot[applied={T|T < 7315382612141260800}]
I20260806 02:08:03.047863 4877 mvcc.cc:205] Tried to move back new op lower bound from 7315382612141260800 to 7315382612029169664. Current Snapshot: MvccSnapshot[applied={T|T < 7315382612141260800}]
I20260806 02:08:03.061056 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:03.057893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:34353 (local address 127.28.21.3:42651)
0806 02:08:03.057999 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:03.058003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:03.058021 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:08:03.058057 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:03.058058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:03.058093 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:03.058178 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:03.058466 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.059114 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.059878 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.060090 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.060166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:03.060598 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:03.060666 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:03.060720 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:03.060859 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 02:08:03.060914 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:08:03.061086 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:03.057401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42651 (local address 127.28.21.1:34353)
0806 02:08:03.057661 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:03.057677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:03.057761 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:03.058189 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:03.058193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:03.058200 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:08:03.058307 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:03.058313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.059214 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.059216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:03.059781 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:03.059789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:03.060350 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:03.060353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:03.060442 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:03.060476 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:08:03.060752 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:08:03.060760 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:08:03.060833 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:08:03.060918 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260806 02:08:03.081046 4283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:03.081224 4283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:03.083244 4283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49167c7be87f4ca4b9d3a8d5111a6da3: Sending DeleteTablet for 3 replicas of tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3
I20260806 02:08:03.083791 4426 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:03 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:60598
I20260806 02:08:03.083910 4578 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:03 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260806 02:08:03.083914 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:03 UTC) from {username='oryx', principal='oryx/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33302
I20260806 02:08:03.083994 4909 tablet_replica.cc:330] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6: stopping tablet replica
I20260806 02:08:03.084123 4909 raft_consensus.cc:2229] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:03.084132 4910 tablet_replica.cc:330] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: stopping tablet replica
I20260806 02:08:03.084245 4909 raft_consensus.cc:2258] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P df355d001a184e6995f18b7ecd7432f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:03.084276 4910 raft_consensus.cc:2229] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:03.084395 4910 raft_consensus.cc:2258] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:03.084393 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 4359
I20260806 02:08:03.084570 4911 tablet_replica.cc:330] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: stopping tablet replica
I20260806 02:08:03.084712 4911 raft_consensus.cc:2229] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:08:03.084911 4911 raft_consensus.cc:2258] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:03.087913 4911 ts_tablet_manager.cc:1905] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:03.090993 4911 ts_tablet_manager.cc:1918] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:08:03.091059 4911 log.cc:1196] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/4ee6e741cbe04a6f9c3fa8afd2c201d3
I20260806 02:08:03.091328 4911 ts_tablet_manager.cc:1939] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 9b114606270d429d8614b4a666c709c2: Deleting consensus metadata
I20260806 02:08:03.091969 4270 catalog_manager.cc:4841] TS 9b114606270d429d8614b4a666c709c2 (127.28.21.3:42651): tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 (table test-table [id=9b71a9860b5d4ced8fa95d9483eea083]) successfully deleted
I20260806 02:08:03.095101 4910 ts_tablet_manager.cc:1905] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:03.097194 4910 ts_tablet_manager.cc:1918] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:08:03.097265 4910 log.cc:1196] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/4ee6e741cbe04a6f9c3fa8afd2c201d3
I20260806 02:08:03.097522 4910 ts_tablet_manager.cc:1939] T 4ee6e741cbe04a6f9c3fa8afd2c201d3 P 11e233df70424577b62eb293e60b3c9e: Deleting consensus metadata
I20260806 02:08:03.098248 4270 catalog_manager.cc:4841] TS 11e233df70424577b62eb293e60b3c9e (127.28.21.2:40129): tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3 (table test-table [id=9b71a9860b5d4ced8fa95d9483eea083]) successfully deleted
W20260806 02:08:03.098949 4270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:40909 (error 108)
I20260806 02:08:03.099116 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 4510
I20260806 02:08:03.101239 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:03.100547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:40909 (local address 127.0.0.1:60608)
0806 02:08:03.101010 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:03.101097 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":131,"threads_started":1}
W20260806 02:08:03.101486 4270 catalog_manager.cc:4584] TS df355d001a184e6995f18b7ecd7432f6 (127.28.21.1:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ee6e741cbe04a6f9c3fa8afd2c201d3: Network error: Client connection negotiation failed: client connection to 127.28.21.1:40909: connect: Connection refused (error 111)
I20260806 02:08:03.109071 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 4656
I20260806 02:08:03.116847 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 4250
2026-08-06T02:08:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3676 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:08:03 dist-test-slave-gp6n krb5kdc[4951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:08:03 dist-test-slave-gp6n krb5kdc[4951](info): set up 2 sockets
Aug 06 02:08:03 dist-test-slave-gp6n krb5kdc[4951](info): commencing operation
krb5kdc: starting...
W20260806 02:08:04.190449 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:08:04 dist-test-slave-gp6n krb5kdc[4951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:08:04.320942 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:35533
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:35533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:04.430459 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:04.430728 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:04.430768 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:04.434664 4966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:04.434715 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:04.434733 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:04.434751 4966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:04.434765 4966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:04.439792 4966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:35533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:35533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:04.440696 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:04.441617 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:04.449975 4975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:04.449983 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:04.450130 4966 server_base.cc:1034] running on GCE node
W20260806 02:08:04.450358 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:04.450991 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:04.452037 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:04.453225 4966 hybrid_clock.cc:648] HybridClock initialized: now 1785982084453211 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:08:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1349 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:08:04 dist-test-slave-gp6n krb5kdc[4980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:08:04 dist-test-slave-gp6n krb5kdc[4980](info): set up 2 sockets
Aug 06 02:08:04 dist-test-slave-gp6n krb5kdc[4980](info): commencing operation
krb5kdc: starting...
W20260806 02:08:05.550514 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:08:05 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:08:05.677958 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45513
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:05.787014 4995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:05.787286 4995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:05.787328 4995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:05.791366 4995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:05.791424 4995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:05.791445 4995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:05.791461 4995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:05.791476 4995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:05.796742 4995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:05.797801 4995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:05.798854 4995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:05.807308 5003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:05.807344 4995 server_base.cc:1034] running on GCE node
W20260806 02:08:05.807274 5000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:05.807274 5001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:05.808048 4995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:05.809082 4995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:05.810261 4995 hybrid_clock.cc:648] HybridClock initialized: now 1785982085810228 us; error 49 us; skew 500 ppm
Aug 06 02:08:05 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982085, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:05.812953 4995 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:08:05.813831 4995 webserver.cc:460] Webserver started at http://127.28.21.62:42331/ using document root <none> and password file <none>
I20260806 02:08:05.814376 4995 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:05.814533 4995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:05.814769 4995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:05.816759 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "dc6aeac1f268467280016b3f29aec75b"
format_stamp: "Formatted at 2026-08-06 02:08:05 on dist-test-slave-gp6n"
server_key: "2d0653f60f5f86955a1d0ffb2912b251"
server_key_iv: "4c526d95e5df61551ca40d80efa78168"
server_key_version: "encryptionkey@0"
I20260806 02:08:05.817317 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "dc6aeac1f268467280016b3f29aec75b"
format_stamp: "Formatted at 2026-08-06 02:08:05 on dist-test-slave-gp6n"
server_key: "2d0653f60f5f86955a1d0ffb2912b251"
server_key_iv: "4c526d95e5df61551ca40d80efa78168"
server_key_version: "encryptionkey@0"
I20260806 02:08:05.820922 4995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:08:05.823266 5010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:05.824465 4995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:05.824612 4995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "dc6aeac1f268467280016b3f29aec75b"
format_stamp: "Formatted at 2026-08-06 02:08:05 on dist-test-slave-gp6n"
server_key: "2d0653f60f5f86955a1d0ffb2912b251"
server_key_iv: "4c526d95e5df61551ca40d80efa78168"
server_key_version: "encryptionkey@0"
I20260806 02:08:05.824723 4995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:05.836897 4995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:05.839933 4995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:05.840149 4995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:05.847340 4995 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:45513
I20260806 02:08:05.847362 5065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:45513 every 8 connection(s)
I20260806 02:08:05.848284 4995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:05.851240 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:05.854154 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 4995
I20260806 02:08:05.854298 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:08:05.854642 28756 external_mini_cluster.cc:1428] Setting key 072c79dc2575acbf703725d10338987b
I20260806 02:08:05.857319 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Bootstrap starting.
I20260806 02:08:05.859962 5066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:05.860855 5066 log.cc:824] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:05.863197 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: No bootstrap required, opened a new log
Aug 06 02:08:05 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:08:05.866357 5066 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:05.866643 5066 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:05.866719 5066 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6aeac1f268467280016b3f29aec75b, State: Initialized, Role: FOLLOWER
I20260806 02:08:05.867127 5066 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:05.867277 5066 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:05.867343 5066 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:05.867460 5066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:05.868314 5066 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:05.868651 5066 leader_election.cc:302] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6aeac1f268467280016b3f29aec75b; no voters:
I20260806 02:08:05.868986 5066 leader_election.cc:288] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:08:05.869127 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:05.869388 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 LEADER]: Becoming Leader. State: Replica: dc6aeac1f268467280016b3f29aec75b, State: Running, Role: LEADER
I20260806 02:08:05.869733 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:05.870325 5066 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:05.870441 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:05.856030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.28.21.62:45513)
0806 02:08:05.856453 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:05.856469 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:05.856506 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:08:05.857120 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:05.857135 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:05.857513 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:05.857786 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:05.858723 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:05.859726 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:05.861014 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:05.861675 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:05.861791 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:05.864639 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:05.864647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:05.864665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:05.864699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:05.867154 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:05.867657 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:05.867663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:05.867668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:05.867785 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:05.868101 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:05.868105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:05.868107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:05.868557 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:05.868768 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:05.869018 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:08:05.869478 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260806 02:08:05.872180 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6aeac1f268467280016b3f29aec75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } } }
I20260806 02:08:05.872301 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:05.872956 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6aeac1f268467280016b3f29aec75b. Latest consensus state: current_term: 1 leader_uuid: "dc6aeac1f268467280016b3f29aec75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } } }
I20260806 02:08:05.873431 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: This master's current role is: LEADER
W20260806 02:08:05.875377 5086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:08:05.875449 5086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:08:05.875545 5087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:05.876437 5087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:05.882570 5087 catalog_manager.cc:1347] Generated new cluster ID: a5c34d090b0d4a6eb9a328171878029f
I20260806 02:08:05.882748 5087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:05.901641 5087 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:08:05.902972 5087 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:05.909446 5087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Generated new TSK 0
I20260806 02:08:05.910176 5087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:08:05.941582 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:06.080056 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:06.080317 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:06.080359 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:06.084471 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:06.084538 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:06.084597 5093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:06.089917 5093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:06.091056 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:06.092150 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:08:06.102037 5093 server_base.cc:1034] running on GCE node
W20260806 02:08:06.102105 5102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.102185 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.102118 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.102730 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:06.103389 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:06.104552 5093 hybrid_clock.cc:648] HybridClock initialized: now 1785982086104536 us; error 25 us; skew 500 ppm
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:06.107368 5093 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:08:06.108299 5093 webserver.cc:460] Webserver started at http://127.28.21.1:39733/ using document root <none> and password file <none>
I20260806 02:08:06.108863 5093 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:06.108953 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:06.109157 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:06.111264 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "acee965e389f40ed9b89bf801c3ecf52"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "fb2322c31eb5cf14b2cc42a637b9efa6"
server_key_iv: "5f07b1ed397a7b6b5ad1191275472ffa"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.111842 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "acee965e389f40ed9b89bf801c3ecf52"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "fb2322c31eb5cf14b2cc42a637b9efa6"
server_key_iv: "5f07b1ed397a7b6b5ad1191275472ffa"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.115474 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:08:06.117857 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.119184 5093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:06.119284 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "acee965e389f40ed9b89bf801c3ecf52"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "fb2322c31eb5cf14b2cc42a637b9efa6"
server_key_iv: "5f07b1ed397a7b6b5ad1191275472ffa"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.119369 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:06.141708 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:06.145102 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:06.145260 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:06.145907 5093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:06.147083 5093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:06.147153 5093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.147219 5093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:06.147249 5093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.156136 5093 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:35601
I20260806 02:08:06.156203 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:35601 every 8 connection(s)
I20260806 02:08:06.157084 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:06.159178 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5093
I20260806 02:08:06.159330 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:08:06.159654 28756 external_mini_cluster.cc:1428] Setting key d10908e9349fe53e98e6688c1d93c58c
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:08:06.172152 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.158974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:49103 (local address 127.28.21.62:45513)
0806 02:08:06.159147 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:06.159154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:06.160091 (+ 937us) server_negotiation.cc:407] Connection header received
0806 02:08:06.161494 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:06.161498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:06.161554 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:06.161655 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:06.162876 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.163542 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.164985 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.165426 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.165568 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.168127 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:06.168133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:06.168137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:06.168169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:06.169973 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.170541 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.170548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.170553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.170652 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.171030 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.171032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.171034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.171215 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:06.171320 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:06.171915 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 02:08:06.172025 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:08:06.172991 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.159429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:49103)
0806 02:08:06.159933 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:06.159988 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:06.161290 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:06.161811 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:06.161818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:06.162164 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:06.162735 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.162749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.163678 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.163685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.164857 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.164867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.165580 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.165583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.165723 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.166448 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:06.166469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:06.167953 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:06.170075 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.170079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.170108 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.170452 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.170747 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.170749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.170751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.170913 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.171302 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:06.171307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:06.171688 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 02:08:06.171842 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 02:08:06.172092 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:08:06.174243 5237 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45513
I20260806 02:08:06.174573 5237 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:06.175060 5237 heartbeater.cc:511] Master 127.28.21.62:45513 requested a full tablet report, sending...
I20260806 02:08:06.176663 5027 ts_manager.cc:194] Registered new tserver with Master: acee965e389f40ed9b89bf801c3ecf52 (127.28.21.1:35601)
I20260806 02:08:06.178339 5027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:49103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:08:06.188922 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:06.303963 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:06.304247 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:06.304322 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:06.308328 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:06.308408 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:06.308493 5242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:06.313637 5242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:06.314845 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:06.315869 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:06.326884 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.326839 5247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.327062 5242 server_base.cc:1034] running on GCE node
W20260806 02:08:06.326853 5248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.327571 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:06.328236 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:06.329443 5242 hybrid_clock.cc:648] HybridClock initialized: now 1785982086329432 us; error 32 us; skew 500 ppm
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:06.332254 5242 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:08:06.333195 5242 webserver.cc:460] Webserver started at http://127.28.21.2:34613/ using document root <none> and password file <none>
I20260806 02:08:06.333765 5242 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:06.333858 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:06.334103 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:06.336251 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "26027ee0498f42f2931b9032ac854016"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "0b71c57b0aa00e265f848f5d552f5636"
server_key_iv: "fdcb02549b959c2001970874953613c9"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.336824 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "26027ee0498f42f2931b9032ac854016"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "0b71c57b0aa00e265f848f5d552f5636"
server_key_iv: "fdcb02549b959c2001970874953613c9"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.340672 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:08:06.343272 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.344475 5242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:06.344614 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "26027ee0498f42f2931b9032ac854016"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "0b71c57b0aa00e265f848f5d552f5636"
server_key_iv: "fdcb02549b959c2001970874953613c9"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.344729 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:06.358143 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:06.361222 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:06.361436 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:06.362030 5242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:06.362917 5242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:06.362991 5242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.363067 5242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:06.363121 5242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.372151 5242 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:32771
I20260806 02:08:06.372202 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:32771 every 8 connection(s)
I20260806 02:08:06.373154 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:06.377774 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5242
I20260806 02:08:06.377933 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:08:06.378223 28756 external_mini_cluster.cc:1428] Setting key 215bef51208a240c75aea5777f057c1c
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:08:06.389838 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.374984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:39975 (local address 127.28.21.62:45513)
0806 02:08:06.375111 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:06.375115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:06.376034 (+ 919us) server_negotiation.cc:407] Connection header received
0806 02:08:06.377427 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:06.377430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:06.377494 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:06.377571 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:06.379009 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.379570 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.382714 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.382956 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.383075 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.385962 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:06.385965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:06.385967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:06.385994 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:06.387625 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.388200 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.388204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.388206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.388292 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.388637 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.388642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.388643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.388840 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:06.388951 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:06.389518 (+ 567us) server_negotiation.cc:299] Negotiation successful
0806 02:08:06.389662 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:08:06.390878 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.375367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:39975)
0806 02:08:06.375791 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:06.375832 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:06.377206 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:06.377741 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:06.377747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:06.378196 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:06.378853 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.378881 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.381602 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.381605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.382586 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.382597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.383117 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.383122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.383293 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.383970 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:06.383998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:06.385771 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:06.387754 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.387761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.387777 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.388091 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.388388 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.388391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.388393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.388539 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.388965 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:06.388973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:06.389264 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 02:08:06.389478 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 02:08:06.389823 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:08:06.392213 5375 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45513
I20260806 02:08:06.392460 5375 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:06.393000 5375 heartbeater.cc:511] Master 127.28.21.62:45513 requested a full tablet report, sending...
I20260806 02:08:06.394093 5027 ts_manager.cc:194] Registered new tserver with Master: 26027ee0498f42f2931b9032ac854016 (127.28.21.2:32771)
I20260806 02:08:06.395006 5027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:39975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:08:06.410617 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:06.521551 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:06.521795 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:06.521836 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:06.526064 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:06.526127 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:06.526185 5380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:06.531528 5380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:06.532451 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:06.533425 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:08:06.543102 5380 server_base.cc:1034] running on GCE node
W20260806 02:08:06.543040 5386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.543069 5388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.543040 5390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.543684 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:06.544288 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:06.545460 5380 hybrid_clock.cc:648] HybridClock initialized: now 1785982086545439 us; error 30 us; skew 500 ppm
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:06.548308 5380 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:08:06.549242 5380 webserver.cc:460] Webserver started at http://127.28.21.3:38633/ using document root <none> and password file <none>
I20260806 02:08:06.549801 5380 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:06.549862 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:06.550098 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:06.552142 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "9023c8a2670340bfba993e6da87aeabf"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "540a323ead9f49a4146e0db690cf8b5f"
server_key_iv: "cea8ddcf3c3f30f47fc9b870b25f8ffe"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.552690 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "9023c8a2670340bfba993e6da87aeabf"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "540a323ead9f49a4146e0db690cf8b5f"
server_key_iv: "cea8ddcf3c3f30f47fc9b870b25f8ffe"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.556334 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:08:06.558740 5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.559926 5380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:08:06.560065 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "9023c8a2670340bfba993e6da87aeabf"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "540a323ead9f49a4146e0db690cf8b5f"
server_key_iv: "cea8ddcf3c3f30f47fc9b870b25f8ffe"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.560170 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:06.583192 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:06.586588 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:06.586787 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:06.587447 5380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:06.588259 5380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:06.588330 5380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.588405 5380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:06.588459 5380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.597121 5380 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:38881
I20260806 02:08:06.597148 5520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:38881 every 8 connection(s)
I20260806 02:08:06.598037 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:08:06.607589 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5380
I20260806 02:08:06.607760 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:08:06.608126 28756 external_mini_cluster.cc:1428] Setting key 7e20181487b5638e3e44279cbae5a175
I20260806 02:08:06.612450 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.599874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:40825 (local address 127.28.21.62:45513)
0806 02:08:06.600017 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:06.600020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:06.600817 (+ 797us) server_negotiation.cc:407] Connection header received
0806 02:08:06.602177 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:06.602180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:06.602241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:06.602348 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:06.603537 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.604086 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.605235 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.605465 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.605577 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.608242 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:06.608246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:06.608248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:06.608277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:06.610030 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.610747 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.610750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.610752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.610844 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:06.611235 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:06.611238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:06.611240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:06.611458 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:06.611579 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:06.612111 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 02:08:06.612252 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:08:06.613322 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:06.600234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:40825)
0806 02:08:06.600663 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:06.600705 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:06.601957 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:06.602550 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:06.602558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:06.602913 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:06.603379 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.603393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.604213 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.604216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.605106 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:06.605115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:06.605616 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:06.605620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:06.605790 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:06.606454 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:06.606472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:06.608063 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:06.610156 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.610161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.610173 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.610616 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.610959 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:06.610962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:06.610964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:06.611118 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:06.611575 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:06.611581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:06.611849 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 02:08:06.612048 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:08:06.612415 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260806 02:08:06.614481 5521 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45513
I20260806 02:08:06.614733 5521 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:06.615213 5521 heartbeater.cc:511] Master 127.28.21.62:45513 requested a full tablet report, sending...
I20260806 02:08:06.616223 5027 ts_manager.cc:194] Registered new tserver with Master: 9023c8a2670340bfba993e6da87aeabf (127.28.21.3:38881)
I20260806 02:08:06.616834 5027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:40825
I20260806 02:08:06.622239 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
Entry for principal oryx/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab.
WARNING: no policy specified for oryx/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
Entry for principal oryx/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab.
WARNING: no policy specified for oryx/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
Entry for principal oryx/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab.
I20260806 02:08:06.699338 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5093
I20260806 02:08:06.706270 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5242
I20260806 02:08:06.712493 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5380
I20260806 02:08:06.718816 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 4995
I20260806 02:08:06.725526 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45513
--webserver_interface=127.28.21.62
--webserver_port=42331
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:06.835438 5531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:06.835754 5531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:06.835837 5531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:06.839828 5531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:06.839922 5531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:06.839962 5531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:06.840025 5531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:06.840062 5531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:06.845376 5531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:06.846488 5531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:06.847533 5531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:06.856360 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.856462 5531 server_base.cc:1034] running on GCE node
W20260806 02:08:06.856366 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:06.856360 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:06.857175 5531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:06.858322 5531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:06.859521 5531 hybrid_clock.cc:648] HybridClock initialized: now 1785982086859487 us; error 47 us; skew 500 ppm
Aug 06 02:08:06 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982086, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:06.862327 5531 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:08:06.863272 5531 webserver.cc:460] Webserver started at http://127.28.21.62:42331/ using document root <none> and password file <none>
I20260806 02:08:06.863843 5531 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:06.863961 5531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:06.867769 5531 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 02:08:06.869850 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:06.871073 5531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:06.871209 5531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "dc6aeac1f268467280016b3f29aec75b"
format_stamp: "Formatted at 2026-08-06 02:08:05 on dist-test-slave-gp6n"
server_key: "2d0653f60f5f86955a1d0ffb2912b251"
server_key_iv: "4c526d95e5df61551ca40d80efa78168"
server_key_version: "encryptionkey@0"
I20260806 02:08:06.871803 5531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:06.896296 5531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:06.899375 5531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:06.899591 5531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:06.906977 5531 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:45513
I20260806 02:08:06.906993 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:45513 every 8 connection(s)
I20260806 02:08:06.908070 5531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:06.912298 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5531
I20260806 02:08:06.912439 5600 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:08:06.912801 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:35601
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:45513
--webserver_port=39733
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:08:06.914172 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Bootstrap starting.
I20260806 02:08:06.937124 5600 log.cc:824] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:06.945500 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:08:06.945935 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Bootstrap complete.
I20260806 02:08:06.951776 5600 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:06.952284 5600 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6aeac1f268467280016b3f29aec75b, State: Initialized, Role: FOLLOWER
I20260806 02:08:06.952828 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:06.952997 5600 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:06.953069 5600 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:06.953166 5600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:08:06.955672 5600 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:06.956033 5600 leader_election.cc:302] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6aeac1f268467280016b3f29aec75b; no voters:
I20260806 02:08:06.956415 5600 leader_election.cc:288] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:08:06.956550 5604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:08:06.956837 5604 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [term 2 LEADER]: Becoming Leader. State: Replica: dc6aeac1f268467280016b3f29aec75b, State: Running, Role: LEADER
I20260806 02:08:06.957144 5604 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } }
I20260806 02:08:06.957800 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:06.958773 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc6aeac1f268467280016b3f29aec75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } } }
I20260806 02:08:06.958887 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:06.958746 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6aeac1f268467280016b3f29aec75b. Latest consensus state: current_term: 2 leader_uuid: "dc6aeac1f268467280016b3f29aec75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6aeac1f268467280016b3f29aec75b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45513 } } }
I20260806 02:08:06.958984 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:06.959245 5613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:06.962797 5613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:06.963445 5613 catalog_manager.cc:1259] Loaded cluster ID: a5c34d090b0d4a6eb9a328171878029f
I20260806 02:08:06.963505 5613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:06.965843 5613 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:06.966539 5613 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc6aeac1f268467280016b3f29aec75b: Loaded TSK: 0
I20260806 02:08:06.967747 5613 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:08:07.042075 5602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:07.042363 5602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:07.042486 5602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:07.046658 5602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:07.046731 5602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:07.046830 5602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:07.052471 5602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.1.keytab
--principal=oryx/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:35601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=39733
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:07.053536 5602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:07.054999 5602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:07.064917 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:07.064944 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:07.065305 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:07.065363 5602 server_base.cc:1034] running on GCE node
I20260806 02:08:07.065747 5602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:07.066475 5602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:07.067690 5602 hybrid_clock.cc:648] HybridClock initialized: now 1785982087067658 us; error 46 us; skew 500 ppm
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982087, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:07.070907 5602 init.cc:377] Logged in from keytab as oryx/127.28.21.1@KRBTEST.COM (short username oryx)
I20260806 02:08:07.072139 5602 webserver.cc:460] Webserver started at http://127.28.21.1:39733/ using document root <none> and password file <none>
I20260806 02:08:07.072974 5602 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:07.073108 5602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:07.078737 5602 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:08:07.081558 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.083231 5602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:08:07.083366 5602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "acee965e389f40ed9b89bf801c3ecf52"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "fb2322c31eb5cf14b2cc42a637b9efa6"
server_key_iv: "5f07b1ed397a7b6b5ad1191275472ffa"
server_key_version: "encryptionkey@0"
I20260806 02:08:07.084132 5602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:07.117509 5602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:07.121438 5602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:07.121608 5602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:07.122195 5602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:07.123133 5602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:07.123188 5602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.123231 5602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:07.123248 5602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.133131 5602 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:35601
I20260806 02:08:07.133160 5755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:35601 every 8 connection(s)
I20260806 02:08:07.134135 5602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:07.141140 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5602
I20260806 02:08:07.141610 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:32771
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:45513
--webserver_port=34613
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.148726 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.136490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:40083)
0806 02:08:07.136984 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.137026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.138232 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.139361 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.139367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.139724 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.140168 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.140181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.141677 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.141681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.143176 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.143191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.144416 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.144419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.144567 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.145193 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.145210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.147965 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
W20260806 02:08:07.149260 5756 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.21.62:45513: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
W20260806 02:08:07.150192 5759 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.136330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:40083 (local address 127.28.21.62:45513)
0806 02:08:07.136815 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.136838 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.137146 (+ 308us) server_negotiation.cc:407] Connection header received
0806 02:08:07.138671 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.138683 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.138991 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.139279 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.140351 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.141535 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.143397 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.144233 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.144439 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.148302 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:40083: BlockingRecv error: recv got EOF from 127.28.21.1:40083 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.155367 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.149796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:60425)
0806 02:08:07.149895 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.149911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.149988 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.150453 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.150456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.150608 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.150707 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.150712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.151491 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.151496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.152055 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.152062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.152533 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.152537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.152628 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.153190 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.153200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.155213 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 02:08:07.155479 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.149893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:60425 (local address 127.28.21.62:45513)
0806 02:08:07.150180 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.150186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.150197 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:08:07.150230 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.150232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.150297 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.150798 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.150877 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.151383 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.152179 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.152414 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.152526 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.155343 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:60425: BlockingRecv error: recv got EOF from 127.28.21.1:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.161307 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.155937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:32775)
0806 02:08:07.156101 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.156112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.156167 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.156402 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.156404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.156530 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.156601 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.156605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.157672 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.157675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.158193 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.158201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.158817 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.158820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.158880 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.159388 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.159399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.161211 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260806 02:08:07.161538 5756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:08:07.161743 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.155985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:32775 (local address 127.28.21.62:45513)
0806 02:08:07.156121 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.156125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.156161 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:08:07.156250 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.156254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.156299 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.156381 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.156725 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.157574 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.158293 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.158712 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.158815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.161408 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:32775: BlockingRecv error: recv got EOF from 127.28.21.1:32775 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:08:07.261606 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:07.261870 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:07.261919 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:07.266551 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:07.266613 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:07.266677 5761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:07.272187 5761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.2.keytab
--principal=oryx/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=34613
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:07.273260 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:07.274329 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:07.283944 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:07.283952 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:07.283952 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:07.284045 5761 server_base.cc:1034] running on GCE node
I20260806 02:08:07.284515 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:07.285115 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:07.286305 5761 hybrid_clock.cc:648] HybridClock initialized: now 1785982087286281 us; error 29 us; skew 500 ppm
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982087, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:07.289431 5761 init.cc:377] Logged in from keytab as oryx/127.28.21.2@KRBTEST.COM (short username oryx)
I20260806 02:08:07.290560 5761 webserver.cc:460] Webserver started at http://127.28.21.2:34613/ using document root <none> and password file <none>
I20260806 02:08:07.291174 5761 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:07.291299 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:07.295332 5761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:08:07.297472 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.298739 5761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:07.298882 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "26027ee0498f42f2931b9032ac854016"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "0b71c57b0aa00e265f848f5d552f5636"
server_key_iv: "fdcb02549b959c2001970874953613c9"
server_key_version: "encryptionkey@0"
I20260806 02:08:07.299526 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:07.312886 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:07.315748 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:07.315965 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:07.316532 5761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:07.317334 5761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:07.317405 5761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.317472 5761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:07.317523 5761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.327848 5761 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:32771
I20260806 02:08:07.327872 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:32771 every 8 connection(s)
I20260806 02:08:07.328855 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:07.329396 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5761
I20260806 02:08:07.329931 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:38881
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:45513
--webserver_port=38633
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:08:07.340788 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.331027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:47651 (local address 127.28.21.62:45513)
0806 02:08:07.331200 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.331207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.332050 (+ 843us) server_negotiation.cc:407] Connection header received
0806 02:08:07.333450 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.333453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.333516 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.333620 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.334962 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.335498 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.336659 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.336897 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.337015 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.340656 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:47651: BlockingRecv error: recv got EOF from 127.28.21.2:47651 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:08:07.341248 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.331410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:47651)
0806 02:08:07.331880 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.331932 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.333223 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.333804 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.333811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.334261 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.334814 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.334828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.335627 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.335637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.336517 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.336526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.337053 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.337058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.337224 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.337836 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.337853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.340439 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":152,"threads_started":1}
W20260806 02:08:07.341750 5898 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.21.62:45513: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.347631 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.342154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:38825)
0806 02:08:07.342351 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.342364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.342495 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.342733 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.342735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.342869 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.342951 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.342956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.343942 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.343944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.344550 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.344559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.345102 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.345105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.345180 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.345707 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.345718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.347536 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 02:08:07.347973 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.342276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:38825 (local address 127.28.21.62:45513)
0806 02:08:07.342397 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.342434 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.342449 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:08:07.342579 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.342582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.342626 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.342708 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.343054 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.343835 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.344657 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.344998 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.345094 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.347693 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:38825: BlockingRecv error: recv got EOF from 127.28.21.2:38825 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.352907 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.348138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:51791)
0806 02:08:07.348291 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.348302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.348389 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.348630 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.348633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.348779 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.348842 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.348846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.349573 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.349578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.350097 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.350104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.350609 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.350613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.350690 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.351147 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.351158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.352820 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260806 02:08:07.353143 5898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:08:07.353055 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.348298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:51791 (local address 127.28.21.62:45513)
0806 02:08:07.348431 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.348438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.348454 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:08:07.348489 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.348492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.348533 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.348617 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.348970 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.349466 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.350219 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.350503 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.350608 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.352960 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:51791: BlockingRecv error: recv got EOF from 127.28.21.2:51791 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:08:07.443234 5900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:07.443478 5900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:07.443521 5900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:07.447427 5900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:07.447484 5900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:07.447543 5900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:07.452641 5900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/krb5kdc/oryx_127.28.21.3.keytab
--principal=oryx/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=38633
--tserver_master_addrs=127.28.21.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:07.453732 5900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:07.454736 5900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:07.464596 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:07.464676 5900 server_base.cc:1034] running on GCE node
W20260806 02:08:07.464612 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:07.464622 5907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:07.465387 5900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:07.466141 5900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:07.467384 5900 hybrid_clock.cc:648] HybridClock initialized: now 1785982087467354 us; error 42 us; skew 500 ppm
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982087, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:07.470501 5900 init.cc:377] Logged in from keytab as oryx/127.28.21.3@KRBTEST.COM (short username oryx)
I20260806 02:08:07.471431 5900 webserver.cc:460] Webserver started at http://127.28.21.3:38633/ using document root <none> and password file <none>
I20260806 02:08:07.472030 5900 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:07.472155 5900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:07.476295 5900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:08:07.478333 5916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.479614 5900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:07.479770 5900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "9023c8a2670340bfba993e6da87aeabf"
format_stamp: "Formatted at 2026-08-06 02:08:06 on dist-test-slave-gp6n"
server_key: "540a323ead9f49a4146e0db690cf8b5f"
server_key_iv: "cea8ddcf3c3f30f47fc9b870b25f8ffe"
server_key_version: "encryptionkey@0"
I20260806 02:08:07.480381 5900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:07.513232 5900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:07.517055 5900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:07.517277 5900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:07.517936 5900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:07.518896 5900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:07.518975 5900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.519061 5900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:07.519109 5900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:07.528692 5900 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:38881
I20260806 02:08:07.528719 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:38881 every 8 connection(s)
I20260806 02:08:07.529788 5900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:08:07.537307 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 5900
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:08:07.541112 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.531719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59007 (local address 127.28.21.62:45513)
0806 02:08:07.531870 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.531873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.532741 (+ 868us) server_negotiation.cc:407] Connection header received
0806 02:08:07.534086 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.534090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.534145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.534233 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.535449 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.535973 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.537123 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.537354 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.537476 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.540981 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:59007: BlockingRecv error: recv got EOF from 127.28.21.3:59007 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:08:07.541738 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.532122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:59007)
0806 02:08:07.532589 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.532631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.533866 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.534473 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.534479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.534839 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.535301 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.535314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.536109 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.536113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.536993 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.537002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.537533 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.537537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.537706 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.538293 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.538309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.540875 (+ 2566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
W20260806 02:08:07.542207 6034 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.21.62:45513: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.547423 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.542837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:35269)
0806 02:08:07.542941 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.542954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.543028 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.543246 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.543249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.543393 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.543472 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.543476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.544166 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.544170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.544699 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.544707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.545138 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.545142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.545230 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.545662 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.545674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.547294 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 02:08:07.547443 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.538456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.28.21.62:45513)
0806 02:08:07.538603 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.538608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.538867 (+ 259us) server_negotiation.cc:407] Connection header received
0806 02:08:07.539043 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.539045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.539098 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.539218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.540040 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.540654 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.541624 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.541880 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.541983 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.543490 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:07.543496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:07.543508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:07.543533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:07.545248 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:07.545705 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:07.545713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:07.545723 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:07.545842 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:07.546126 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:07.546129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:07.546131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:07.546572 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:07.546772 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:07.547068 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 02:08:07.547306 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:08:07.547587 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.542940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:35269 (local address 127.28.21.62:45513)
0806 02:08:07.543053 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.543060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.543074 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:07.543109 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.543111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.543151 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.543233 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.543589 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.544062 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.544816 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.545031 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.545137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.547402 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:35269: BlockingRecv error: recv got EOF from 127.28.21.3:35269 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:07 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:07.553020 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:07.547963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:49789)
0806 02:08:07.548066 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:07.548079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:07.548143 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:07.548700 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:07.548702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:07.548847 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:07.548925 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.548931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.550020 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.550023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.550590 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:07.550598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.550978 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.550981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:07.551036 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.551449 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:07.551460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:07.552938 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:08:07.553217 6034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:08:07.553225 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:07.548375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:49789 (local address 127.28.21.62:45513)
0806 02:08:07.548505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:07.548509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:07.548521 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:08:07.548559 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:07.548561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:07.548590 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:07.548652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:07.549032 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.549915 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.550686 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:07.550889 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:07.550966 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:07.553112 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:49789: BlockingRecv error: recv got EOF from 127.28.21.3:49789 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:08.168102 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:08.162199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:32827)
0806 02:08:08.162498 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:08.162514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:08.162605 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:08.162858 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:08.162860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:08.163047 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:08.163159 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.163164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.163946 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.163949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.164613 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.164623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.165087 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.165090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.165170 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.165784 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:08.165796 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:08.167907 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260806 02:08:08.168241 6039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:08.162254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:32827 (local address 127.28.21.62:45513)
0806 02:08:08.162550 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:08.162554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:08.162568 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:08.162693 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:08.162695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:08.162740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:08.162821 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:08.163264 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.163820 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.164731 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.164958 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.165039 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.168074 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:32827: BlockingRecv error: recv got EOF from 127.28.21.1:32827 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:08.359793 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:08.353803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:38249)
0806 02:08:08.354054 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:08.354077 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:08.354177 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:08.354563 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:08.354567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:08.354792 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:08.354915 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.354922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.355699 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.355704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.356324 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.356333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.356858 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.356861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.356967 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.357610 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:08.357623 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:08.359662 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
W20260806 02:08:08.359936 6039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:08.353858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:38249 (local address 127.28.21.62:45513)
0806 02:08:08.354025 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:08.354031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:08.354149 (+ 118us) server_negotiation.cc:407] Connection header received
0806 02:08:08.354249 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:08.354251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:08.354330 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:08.354534 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:08.355045 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.355584 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.356463 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.356725 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.356842 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.359810 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:38249: BlockingRecv error: recv got EOF from 127.28.21.2:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:08 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:08.559731 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:08.553821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:56253)
0806 02:08:08.554106 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:08.554120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:08.554221 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:08.554475 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:08.554477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:08.554672 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:08.554807 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.554813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.555604 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.555608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.556232 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:08.556242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.556724 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.556727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:08.556809 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.557317 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:08.557331 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:08.559549 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
W20260806 02:08:08.559863 6039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:08.553944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:56253 (local address 127.28.21.62:45513)
0806 02:08:08.554086 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:08.554095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:08.554190 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:08:08.554302 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:08.554304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:08.554350 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:08.554446 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:08.554916 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.555472 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.556369 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:08.556603 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:08.556694 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:08.559736 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:56253: BlockingRecv error: recv got EOF from 127.28.21.3:56253 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:09.174789 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:09.168936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:33081)
0806 02:08:09.169213 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:09.169227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:09.169334 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:09.169600 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:09.169603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:09.169788 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:09.169888 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.169893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.170832 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.170837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.171411 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.171419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.171896 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.171900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.171988 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.172506 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:09.172518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:09.174585 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260806 02:08:09.174834 6043 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:09.168968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33081 (local address 127.28.21.62:45513)
0806 02:08:09.169242 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:09.169246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:09.169274 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:08:09.169422 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:09.169424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:09.169469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:09.169554 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:09.170002 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.170725 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.171545 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.171783 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.171890 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.174713 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:33081: BlockingRecv error: recv got EOF from 127.28.21.1:33081 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":124,"threads_started":1}
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:09.365828 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:09.360619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:53473)
0806 02:08:09.360889 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:09.360903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:09.361000 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:09.361243 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:09.361246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:09.361426 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:09.361533 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.361538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.362252 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.362255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.362864 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.362873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.363318 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.363321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.363389 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.363913 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:09.363924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:09.365694 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260806 02:08:09.365983 6043 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:09.360734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:53473 (local address 127.28.21.62:45513)
0806 02:08:09.360874 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:09.360877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:09.360969 (+ 92us) server_negotiation.cc:407] Connection header received
0806 02:08:09.361082 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:09.361084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:09.361130 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:09.361203 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:09.361630 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.362162 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.362991 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.363216 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.363302 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.365856 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:53473: BlockingRecv error: recv got EOF from 127.28.21.2:53473 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:09 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:09.570806 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:09.560616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:54055)
0806 02:08:09.560947 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:09.560966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:09.561069 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:09.561306 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:09.561308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:09.561492 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:09.561591 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.561596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.562301 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.562304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.562894 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:09.562903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.563497 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.563500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:09.563633 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.568475 (+ 4842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:09.568508 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:09.570646 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
W20260806 02:08:09.570919 6043 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:09.560759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:54055 (local address 127.28.21.62:45513)
0806 02:08:09.560906 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:09.560909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:09.561036 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:08:09.561151 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:09.561153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:09.561198 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:09.561266 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:09.561690 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.562207 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.563010 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:09.563341 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:09.563466 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:09.570786 (+ 7320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:54055: BlockingRecv error: recv got EOF from 127.28.21.3:54055 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:10.181231 6046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:10.175675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:54673)
0806 02:08:10.175936 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:10.175952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:10.176057 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:10.176385 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:10.176387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:10.176567 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:10.176663 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.176668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.177471 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.177473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.178058 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.178066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.178546 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.178548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.178612 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.179186 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:10.179206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:10.181115 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":109,"threads_started":1}
W20260806 02:08:10.181453 6047 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:10.175855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:54673 (local address 127.28.21.62:45513)
0806 02:08:10.176141 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:10.176144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:10.176158 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:10.176195 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:10.176197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:10.176237 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:10.176368 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:10.176792 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.177368 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.178167 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.178449 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.178531 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.181290 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:54673: BlockingRecv error: recv got EOF from 127.28.21.1:54673 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":128,"threads_started":1}
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:10.373160 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:10.366741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:34085)
0806 02:08:10.367126 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:10.367143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:10.367322 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:10.367638 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:10.367642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:10.367876 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:10.367976 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.367982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.368846 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.368854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.369639 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.369648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.370249 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.370252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.370355 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.371015 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:10.371030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:10.373027 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
W20260806 02:08:10.373318 6047 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:10.366876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:34085 (local address 127.28.21.62:45513)
0806 02:08:10.367088 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:10.367093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:10.367246 (+ 153us) server_negotiation.cc:407] Connection header received
0806 02:08:10.367448 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:10.367453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:10.367515 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:10.367614 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:10.368109 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.368719 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.369819 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.370119 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.370244 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.373161 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:34085: BlockingRecv error: recv got EOF from 127.28.21.2:34085 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:10 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:10.577235 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:10.571600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:52727)
0806 02:08:10.571913 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:10.571926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:10.572067 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:10.572344 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:10.572347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:10.572562 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:10.572659 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.572664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.573435 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.573440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.574037 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:10.574045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.574542 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.574562 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:10.574655 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.575138 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:10.575150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:10.577099 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
W20260806 02:08:10.577344 6047 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:10.571655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:52727 (local address 127.28.21.62:45513)
0806 02:08:10.571832 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:10.571837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:10.572007 (+ 170us) server_negotiation.cc:407] Connection header received
0806 02:08:10.572170 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:10.572173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:10.572228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:10.572314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:10.572782 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.573321 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.574159 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:10.574385 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:10.574541 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:10.577232 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:52727: BlockingRecv error: recv got EOF from 127.28.21.3:52727 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:11.188059 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:11.182040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:33757)
0806 02:08:11.182447 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:11.182466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:11.182568 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:11.182764 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:11.182766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:11.182953 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:11.183047 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.183051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.183876 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.183880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.184597 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.184608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.185152 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.185156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.185240 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.185785 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:11.185798 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:11.187925 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
W20260806 02:08:11.188197 6051 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:11.182075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33757 (local address 127.28.21.62:45513)
0806 02:08:11.182327 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:11.182334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:11.182532 (+ 198us) server_negotiation.cc:407] Connection header received
0806 02:08:11.182634 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:11.182636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:11.182680 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:11.182756 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:11.183141 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.183698 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.184733 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.185010 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.185094 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.188063 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:33757: BlockingRecv error: recv got EOF from 127.28.21.1:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:11.380084 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:11.373943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:38387)
0806 02:08:11.374225 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:11.374238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:11.374347 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:11.374692 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:11.374694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:11.374886 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:11.374987 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.374992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.375892 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.375895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.376486 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.376494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.377140 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.377145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.377249 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.377910 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:11.377931 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:11.379958 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
W20260806 02:08:11.380302 6051 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:11.374080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:38387 (local address 127.28.21.62:45513)
0806 02:08:11.374241 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:11.374245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:11.374462 (+ 217us) server_negotiation.cc:407] Connection header received
0806 02:08:11.374524 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:11.374526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:11.374574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:11.374648 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:11.375117 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.375783 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.376618 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.376999 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.377138 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.380158 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:38387: BlockingRecv error: recv got EOF from 127.28.21.2:38387 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:11 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:11.584049 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:11.578137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:33545)
0806 02:08:11.578480 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:11.578494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:11.578620 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:11.578896 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:11.578899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:11.579102 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:11.579198 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.579204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.579983 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.579988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.580547 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:11.580554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.581032 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.581035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:11.581122 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.581684 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:11.581704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:11.583895 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
W20260806 02:08:11.584245 6051 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:11.578477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:33545 (local address 127.28.21.62:45513)
0806 02:08:11.578623 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:11.578628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:11.578643 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:08:11.578715 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:11.578719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:11.578768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:11.578868 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:11.579327 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.579869 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.580668 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:11.580917 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:11.581029 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:11.584081 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:33545: BlockingRecv error: recv got EOF from 127.28.21.3:33545 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:12.194617 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:12.188997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:49989)
0806 02:08:12.189317 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:12.189340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:12.189434 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:12.189781 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:12.189783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:12.189977 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:12.190073 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.190078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.190900 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.190903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.191524 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.191533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.191957 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.191960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.192025 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.192603 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:12.192615 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:12.194488 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
W20260806 02:08:12.194758 6056 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:12.189150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:49989 (local address 127.28.21.62:45513)
0806 02:08:12.189394 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:12.189398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:12.189412 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:12.189568 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:12.189572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:12.189628 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:12.189726 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:12.190172 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.190803 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.191624 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.191853 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.191933 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.194597 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:49989: BlockingRecv error: recv got EOF from 127.28.21.1:49989 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:12.386611 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:12.380882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:49737)
0806 02:08:12.381181 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:12.381194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:12.381289 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:12.381518 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:12.381520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:12.381710 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:12.381806 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.381811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.382539 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.382542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.383151 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.383180 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.383746 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.383750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.383851 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.384512 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:12.384529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:12.386392 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":128,"threads_started":1}
W20260806 02:08:12.386797 6056 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:12.381041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:49737 (local address 127.28.21.62:45513)
0806 02:08:12.381179 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:12.381183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:12.381263 (+ 80us) server_negotiation.cc:407] Connection header received
0806 02:08:12.381360 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:12.381362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:12.381403 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:12.381472 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:12.381906 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.382448 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.383293 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.383618 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.383724 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.386634 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:49737: BlockingRecv error: recv got EOF from 127.28.21.2:49737 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:12 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:12.589952 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:12.584817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:59195)
0806 02:08:12.585089 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:12.585102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:12.585191 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:12.585411 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:12.585413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:12.585589 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:12.585681 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.585686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.586389 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.586391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.587021 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:12.587030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.587437 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.587439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:12.587498 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.587948 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:12.587959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:12.589832 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
W20260806 02:08:12.590076 6056 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:12.584863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59195 (local address 127.28.21.62:45513)
0806 02:08:12.585009 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:12.585013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:12.585165 (+ 152us) server_negotiation.cc:407] Connection header received
0806 02:08:12.585265 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:12.585267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:12.585311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:12.585378 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:12.585775 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.586299 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.587122 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:12.587340 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:12.587421 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:12.589963 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:59195: BlockingRecv error: recv got EOF from 127.28.21.3:59195 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:13.201254 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:13.195373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:44971)
0806 02:08:13.195695 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:13.195714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:13.195842 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:13.196221 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:13.196224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:13.196429 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:13.196532 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.196538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.197268 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.197271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.197916 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.197925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.198347 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.198350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.198442 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.198959 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:13.198972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:13.201129 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
W20260806 02:08:13.201433 6061 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:13.195493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:44971 (local address 127.28.21.62:45513)
0806 02:08:13.195732 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:13.195736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:13.195774 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:08:13.196036 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:13.196038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:13.196086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:13.196168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:13.196641 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.197162 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.198027 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.198243 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.198317 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.201286 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:44971: BlockingRecv error: recv got EOF from 127.28.21.1:44971 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:13.393508 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:13.387376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:60503)
0806 02:08:13.387801 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:13.387822 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:13.387977 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:13.388256 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:13.388259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:13.388479 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:13.388574 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.388581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.389410 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.389414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.390020 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.390029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.390550 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.390554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.390643 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.391331 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:13.391345 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:13.393383 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":88,"threads_started":1}
W20260806 02:08:13.393719 6061 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:13.387473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:60503 (local address 127.28.21.62:45513)
0806 02:08:13.387772 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:13.387777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:13.387898 (+ 121us) server_negotiation.cc:407] Connection header received
0806 02:08:13.388099 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:13.388101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:13.388145 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:13.388235 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:13.388690 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.389296 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.390142 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.390374 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.390550 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.393532 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:60503: BlockingRecv error: recv got EOF from 127.28.21.2:60503 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:13 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:13.596616 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:13.590806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:52697)
0806 02:08:13.591100 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:13.591113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:13.591238 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:13.591481 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:13.591483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:13.591667 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:13.591761 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.591766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.592609 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.592611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.593160 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:13.593166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.593664 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.593666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:13.593725 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.594282 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:13.594293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:13.596440 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
W20260806 02:08:13.596848 6061 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:13.590932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:52697 (local address 127.28.21.62:45513)
0806 02:08:13.591074 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:13.591078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:13.591173 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:08:13.591326 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:13.591328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:13.591371 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:13.591441 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:13.591896 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.592504 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.593272 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:13.593551 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:13.593643 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:13.596611 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:52697: BlockingRecv error: recv got EOF from 127.28.21.3:52697 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:14.207540 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:14.202132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:46475)
0806 02:08:14.202457 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:14.202471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:14.202574 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:14.202817 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:14.202819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:14.203086 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:14.203185 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.203210 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.203996 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.204000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.204622 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.204631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.205040 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.205043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.205103 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.205585 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:14.205597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:14.207422 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
W20260806 02:08:14.207746 6065 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:14.202182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:46475 (local address 127.28.21.62:45513)
0806 02:08:14.202483 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:14.202487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:14.202537 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:08:14.202653 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:14.202655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:14.202716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:14.202794 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:14.203325 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.203900 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.204727 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.204944 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.205034 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.207569 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:46475: BlockingRecv error: recv got EOF from 127.28.21.1:46475 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:14.400115 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:14.394337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:50879)
0806 02:08:14.394712 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:14.394727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:14.394876 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:14.395228 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:14.395231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:14.395431 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:14.395528 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.395534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.396322 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.396325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.396878 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.396886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.397331 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.397333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.397398 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.397952 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:14.397974 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:14.399982 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":98,"threads_started":1}
W20260806 02:08:14.400292 6065 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:14.394466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:50879 (local address 127.28.21.62:45513)
0806 02:08:14.394619 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:14.394625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:14.394833 (+ 208us) server_negotiation.cc:407] Connection header received
0806 02:08:14.394982 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:14.394986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:14.395087 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:14.395181 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:14.395637 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.396213 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.396992 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.397233 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.397318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.400130 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:50879: BlockingRecv error: recv got EOF from 127.28.21.2:50879 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:14 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:14.606493 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:14.597368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:32915)
0806 02:08:14.597747 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:14.597763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:14.597886 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:14.598149 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:14.598152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:14.598347 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:14.598525 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.598532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.599266 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.599273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.600158 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:14.600168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.600555 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.600557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:14.600615 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.604375 (+ 3760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:14.604390 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:14.606305 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
W20260806 02:08:14.606571 6065 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:14.597463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:32915 (local address 127.28.21.62:45513)
0806 02:08:14.597612 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:14.597617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:14.597855 (+ 238us) server_negotiation.cc:407] Connection header received
0806 02:08:14.597957 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:14.597959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:14.598015 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:14.598081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:14.598627 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.599164 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.600250 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:14.600467 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:14.600540 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:14.606474 (+ 5934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:32915: BlockingRecv error: recv got EOF from 127.28.21.3:32915 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:15.214144 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:15.208316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:57393)
0806 02:08:15.208654 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:15.208669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:15.208764 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:15.209011 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:15.209013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:15.209201 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:15.209316 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.209321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.210185 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.210189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.210976 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.210986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.211471 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.211474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.211560 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.212052 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:15.212064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:15.214005 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":154,"threads_started":1}
W20260806 02:08:15.214273 6069 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:15.208490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:57393 (local address 127.28.21.62:45513)
0806 02:08:15.208751 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:15.208754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:15.208768 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:15.208849 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:15.208851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:15.208893 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:15.208961 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:15.209427 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.210065 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.211095 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.211345 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.211465 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.214142 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:57393: BlockingRecv error: recv got EOF from 127.28.21.1:57393 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:15.407496 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:15.400877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:37573)
0806 02:08:15.401218 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:15.401236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:15.401403 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:15.401711 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:15.401715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:15.401951 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:15.402050 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.402058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.403299 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.403303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.403939 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.403948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.404547 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.404551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.404640 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.405308 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:15.405324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:15.407317 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
W20260806 02:08:15.407637 6069 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:15.401013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:37573 (local address 127.28.21.62:45513)
0806 02:08:15.401198 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:15.401203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:15.401335 (+ 132us) server_negotiation.cc:407] Connection header received
0806 02:08:15.401516 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:15.401530 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:15.401588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:15.401686 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:15.402185 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.403157 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.404072 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.404420 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.404544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.407512 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:37573: BlockingRecv error: recv got EOF from 127.28.21.2:37573 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:15 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:15.612567 6071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:15.607378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:57645)
0806 02:08:15.607702 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:15.607715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:15.607817 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:15.608065 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:15.608067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:15.608247 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:15.608343 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.608348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.609057 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.609059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.609622 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:15.609630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.610040 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.610042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:15.610101 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.610652 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:15.610664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:15.612447 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
W20260806 02:08:15.612707 6069 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:15.607495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:57645 (local address 127.28.21.62:45513)
0806 02:08:15.607647 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:15.607651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:15.607781 (+ 130us) server_negotiation.cc:407] Connection header received
0806 02:08:15.607896 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:15.607898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:15.607945 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:15.608034 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:15.608444 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.608963 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.609724 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:15.609943 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:15.610021 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:15.612590 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:57645: BlockingRecv error: recv got EOF from 127.28.21.3:57645 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:16.223285 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:16.215075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:58517)
0806 02:08:16.215414 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:16.215452 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:16.215555 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:16.215818 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:16.215820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:16.216073 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:16.216199 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.216209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.217687 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.217693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.218782 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.218797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.219589 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.219595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.219714 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.220390 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:16.220409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:16.223111 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
W20260806 02:08:16.223376 6072 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:16.215135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:58517 (local address 127.28.21.62:45513)
0806 02:08:16.215442 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:16.215446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:16.215592 (+ 146us) server_negotiation.cc:407] Connection header received
0806 02:08:16.215641 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:16.215643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:16.215692 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:16.215785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:16.216368 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.217537 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.218941 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.219448 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.219583 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.223247 (+ 3664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:58517: BlockingRecv error: recv got EOF from 127.28.21.1:58517 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:16.413789 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:16.408243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:34425)
0806 02:08:16.408509 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:16.408522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:16.408622 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:16.408860 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:16.408862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:16.409046 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:16.409142 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.409147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.409930 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.409932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.410508 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.410515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.410941 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.410943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.411008 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.411559 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:16.411570 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:16.413601 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
W20260806 02:08:16.413856 6072 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:16.408325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:34425 (local address 127.28.21.62:45513)
0806 02:08:16.408504 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:16.408510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:16.408600 (+ 90us) server_negotiation.cc:407] Connection header received
0806 02:08:16.408708 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:16.408711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:16.408757 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:16.408828 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:16.409243 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.409832 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.410605 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.410845 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.410939 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.413739 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:34425: BlockingRecv error: recv got EOF from 127.28.21.2:34425 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:16 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:16.618954 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:16.613448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:36195)
0806 02:08:16.613733 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:16.613746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:16.613881 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:16.614159 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:16.614162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:16.614374 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:16.614495 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.614500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.615272 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.615278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.615852 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:16.615861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.616334 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.616338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:16.616427 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.616928 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:16.616940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:16.618766 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
W20260806 02:08:16.618992 6072 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:16.613508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:36195 (local address 127.28.21.62:45513)
0806 02:08:16.613665 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:16.613670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:16.613821 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:08:16.613990 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:16.613994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:16.614046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:16.614134 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:16.614636 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.615165 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.615987 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:16.616215 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:16.616320 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:16.618886 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:36195: BlockingRecv error: recv got EOF from 127.28.21.3:36195 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:17.230315 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:17.224215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:33553)
0806 02:08:17.224503 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:17.224526 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:17.224645 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:17.224975 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:17.224981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:17.225208 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:17.225307 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.225313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.226337 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.226340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.226933 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.226941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.227442 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.227444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.227508 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.228021 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:17.228032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:17.230125 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
W20260806 02:08:17.230511 6078 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:17.224340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33553 (local address 127.28.21.62:45513)
0806 02:08:17.224657 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:17.224661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:17.224677 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:08:17.224722 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:17.224724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:17.224802 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:17.224921 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:17.225436 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.226230 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.227052 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.227298 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.227435 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.230267 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:33553: BlockingRecv error: recv got EOF from 127.28.21.1:33553 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:17.419847 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:17.414662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:47497)
0806 02:08:17.414939 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:17.414954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:17.415049 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:17.415279 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:17.415281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:17.415475 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:17.415573 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.415578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.416287 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.416290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.416864 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.416872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.417271 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.417273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.417333 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.417905 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:17.417917 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:17.419732 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
W20260806 02:08:17.420003 6078 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:17.414724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:47497 (local address 127.28.21.62:45513)
0806 02:08:17.414864 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:17.414868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:17.415023 (+ 155us) server_negotiation.cc:407] Connection header received
0806 02:08:17.415131 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:17.415133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:17.415174 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:17.415240 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:17.415684 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.416199 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.416972 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.417183 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.417255 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.419851 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:47497: BlockingRecv error: recv got EOF from 127.28.21.2:47497 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:17 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:17.625303 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:17.619758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:55955)
0806 02:08:17.620162 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:17.620174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:17.620291 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:17.620578 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:17.620581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:17.620798 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:17.620888 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.620895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.621656 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.621661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.622245 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:17.622252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.622774 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.622778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:17.622866 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.623360 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:17.623373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:17.625189 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":124,"threads_started":1}
W20260806 02:08:17.625442 6078 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:17.619862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:55955 (local address 127.28.21.62:45513)
0806 02:08:17.620021 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:17.620027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:17.620249 (+ 222us) server_negotiation.cc:407] Connection header received
0806 02:08:17.620413 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:17.620420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:17.620466 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:17.620548 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:17.621008 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.621539 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.622367 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:17.622654 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:17.622761 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:17.625322 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:55955: BlockingRecv error: recv got EOF from 127.28.21.3:55955 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:18.236435 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:18.231103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:40251)
0806 02:08:18.231369 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:18.231381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:18.231472 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:18.231839 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:18.231841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:18.232046 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:18.232148 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.232153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.232892 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.232894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.233446 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.233452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.233882 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.233884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.233946 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.234457 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:18.234471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:18.236321 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260806 02:08:18.236656 6082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:18.231140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:40251 (local address 127.28.21.62:45513)
0806 02:08:18.231639 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:18.231642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:18.231652 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:08:18.231682 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:18.231684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:18.231721 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:18.231790 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:18.232262 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.232798 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.233554 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.233781 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.233864 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.236495 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:40251: BlockingRecv error: recv got EOF from 127.28.21.1:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":78,"threads_started":1}
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:18.426471 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:18.420751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:33059)
0806 02:08:18.421067 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:18.421079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:18.421217 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:18.421500 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:18.421502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:18.421715 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:18.421809 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.421814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.422676 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.422681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.423249 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.423257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.423723 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.423726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.423810 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.424382 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:18.424396 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:18.426337 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
W20260806 02:08:18.426678 6082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:18.420831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:33059 (local address 127.28.21.62:45513)
0806 02:08:18.421011 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:18.421017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:18.421156 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:08:18.421321 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:18.421325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:18.421377 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:18.421475 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:18.421952 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.422570 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.423372 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.423607 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.423719 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.426552 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:33059: BlockingRecv error: recv got EOF from 127.28.21.2:33059 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:18 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:18.631687 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:18.626133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:53405)
0806 02:08:18.626466 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:18.626483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:18.626622 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:18.626918 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:18.626921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:18.627133 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:18.627225 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.627231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.627994 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.627999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.628563 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:18.628571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.629037 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.629041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:18.629123 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.629696 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:18.629708 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:18.631574 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
W20260806 02:08:18.631875 6082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:18.626212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:53405 (local address 127.28.21.62:45513)
0806 02:08:18.626376 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:18.626380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:18.626564 (+ 184us) server_negotiation.cc:407] Connection header received
0806 02:08:18.626726 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:18.626731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:18.626784 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:18.626870 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:18.627351 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.627883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.628684 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:18.628916 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:18.629023 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:18.631713 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:53405: BlockingRecv error: recv got EOF from 127.28.21.3:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:19.242759 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:19.237193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:53141)
0806 02:08:19.237503 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:19.237518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:19.237611 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:19.237869 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:19.237872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:19.238067 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:19.238166 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.238171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.238973 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.238976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.239551 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.239563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.239971 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.239973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.240040 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.240631 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:19.240643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:19.242590 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
W20260806 02:08:19.242833 6087 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:19.237287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:53141 (local address 127.28.21.62:45513)
0806 02:08:19.237553 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:19.237556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:19.237570 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:08:19.237704 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:19.237706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:19.237756 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:19.237832 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:19.238265 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.238857 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.239655 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.239874 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.239952 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.242708 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:53141: BlockingRecv error: recv got EOF from 127.28.21.1:53141 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:19.432924 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:19.427257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:59299)
0806 02:08:19.427543 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:19.427558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:19.427701 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:19.427968 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:19.427972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:19.428192 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:19.428281 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.428286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.429038 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.429041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.429650 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.429660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.430258 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.430262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.430348 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.430923 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:19.430945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:19.432757 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
W20260806 02:08:19.432996 6087 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:19.427343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:59299 (local address 127.28.21.62:45513)
0806 02:08:19.427520 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:19.427525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:19.427637 (+ 112us) server_negotiation.cc:407] Connection header received
0806 02:08:19.427802 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:19.427805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:19.427858 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:19.427939 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:19.428392 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.428926 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.429768 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.430139 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.430255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.432876 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:59299: BlockingRecv error: recv got EOF from 127.28.21.2:59299 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:19 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:19.642889 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:19.632519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:51109)
0806 02:08:19.632785 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:19.632799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:19.632911 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:19.633175 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:19.633177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:19.633362 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:19.633467 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.633472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.634190 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.634192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.634806 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:19.634814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.635326 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.635330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:19.635401 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.640689 (+ 5288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:19.640701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:19.642770 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
W20260806 02:08:19.643034 6087 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:19.632570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:51109 (local address 127.28.21.62:45513)
0806 02:08:19.632705 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:19.632708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:19.632868 (+ 160us) server_negotiation.cc:407] Connection header received
0806 02:08:19.632993 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:19.633002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:19.633048 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:19.633118 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:19.633569 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.634090 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.634955 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:19.635193 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:19.635288 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:19.642901 (+ 7613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:51109: BlockingRecv error: recv got EOF from 127.28.21.3:51109 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:20.249065 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:20.243579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:49395)
0806 02:08:20.243870 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:20.243884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:20.243971 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:20.244220 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:20.244222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:20.244401 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:20.244492 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.244498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.245324 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.245327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.245878 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.245885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.246362 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.246364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.246493 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.246994 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:20.247014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:20.248889 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
W20260806 02:08:20.249228 6091 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:20.243638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:49395 (local address 127.28.21.62:45513)
0806 02:08:20.243934 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:20.243938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:20.243953 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:08:20.244071 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:20.244074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:20.244122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:20.244211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:20.244603 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.245223 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.246005 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.246252 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.246394 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.249087 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:49395: BlockingRecv error: recv got EOF from 127.28.21.1:49395 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:20.439721 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:20.433911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:48623)
0806 02:08:20.434316 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:20.434328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:20.434519 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:20.434832 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:20.434836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:20.435097 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:20.435224 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.435230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.435960 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.435962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.436516 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.436522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.436992 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.437007 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.437067 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.437582 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:20.437593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:20.439586 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":142,"threads_started":1}
W20260806 02:08:20.440009 6091 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:20.434022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:48623 (local address 127.28.21.62:45513)
0806 02:08:20.434190 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:20.434197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:20.434428 (+ 231us) server_negotiation.cc:407] Connection header received
0806 02:08:20.434635 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:20.434638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:20.434701 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:20.434790 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:20.435339 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.435871 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.436645 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.436891 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.436983 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.439753 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:48623: BlockingRecv error: recv got EOF from 127.28.21.2:48623 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:20 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:20.649587 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:20.643859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:40963)
0806 02:08:20.644197 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:20.644211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:20.644337 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:20.644694 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:20.644697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:20.644921 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:20.645013 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.645019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.645806 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.645811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.646452 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:20.646466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.646922 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.646926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:20.647010 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.647594 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:20.647615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:20.649392 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":91,"threads_started":1}
W20260806 02:08:20.649655 6091 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:20.643885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:40963 (local address 127.28.21.62:45513)
0806 02:08:20.644043 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:20.644048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:20.644285 (+ 237us) server_negotiation.cc:407] Connection header received
0806 02:08:20.644473 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:20.644477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:20.644552 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:20.644666 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:20.645150 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.645692 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.646569 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:20.646803 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:20.646917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:20.649539 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:40963: BlockingRecv error: recv got EOF from 127.28.21.3:40963 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:21.256502 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:21.249995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:49577)
0806 02:08:21.250335 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:21.250350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:21.250482 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:21.250754 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:21.250757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:21.251001 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:21.251109 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.251115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.251970 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.251973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.253114 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.253125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.253639 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.253642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.253718 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.254361 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:21.254388 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:21.256365 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
W20260806 02:08:21.256840 6095 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:21.250095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:49577 (local address 127.28.21.62:45513)
0806 02:08:21.250385 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:21.250389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:21.250445 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:08:21.250567 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:21.250569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:21.250613 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:21.250680 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:21.251223 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.251742 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.253257 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.253498 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.253589 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.256544 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:49577: BlockingRecv error: recv got EOF from 127.28.21.1:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":59,"threads_started":1}
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:21.446058 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:21.440515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:41165)
0806 02:08:21.440802 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:21.440815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:21.440916 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:21.441208 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:21.441210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:21.441393 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:21.441485 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.441489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.442266 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.442270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.443034 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.443048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.443558 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.443561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.443653 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.444164 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:21.444175 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:21.445937 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
W20260806 02:08:21.446192 6095 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:21.440652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:41165 (local address 127.28.21.62:45513)
0806 02:08:21.440809 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:21.440813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:21.440894 (+ 81us) server_negotiation.cc:407] Connection header received
0806 02:08:21.441036 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:21.441038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:21.441090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:21.441172 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:21.441606 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.442158 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.443149 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.443465 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.443546 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.446073 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:41165: BlockingRecv error: recv got EOF from 127.28.21.2:41165 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:21 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.3@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:21.656425 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:21.650393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.3:56201)
0806 02:08:21.650750 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:21.650763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:21.650896 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:21.651225 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:21.651228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:21.651482 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:21.651614 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.651620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.652450 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.652452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.653019 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:21.653026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.653509 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.653512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:21.653572 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.654123 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:21.654134 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:21.656232 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":131,"threads_started":1}
W20260806 02:08:21.656477 6095 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:21.650537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:56201 (local address 127.28.21.62:45513)
0806 02:08:21.650700 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:21.650705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:21.650873 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:08:21.651019 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:21.651022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:21.651089 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:21.651184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:21.651728 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.652349 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.653141 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:21.653398 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:21.653491 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:21.656364 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.3:56201: BlockingRecv error: recv got EOF from 127.28.21.3:56201 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 02:08:22 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:22 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.1@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:22.262843 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:22.257357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.1:57227)
0806 02:08:22.257674 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:22.257688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:22.257774 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:22.258013 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:22.258015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:22.258196 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:22.258292 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:22.258296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.259160 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.259163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:22.259823 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:22.259832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.260269 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.260272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:22.260336 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:22.260899 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:22.260911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:22.262731 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
W20260806 02:08:22.262992 6099 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:22.257548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:57227 (local address 127.28.21.62:45513)
0806 02:08:22.257747 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:22.257750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:22.257763 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:08:22.257863 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:22.257865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:22.257905 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:22.257979 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:22.258427 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.259061 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.259937 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.260176 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.260252 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:22.262857 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.1:57227: BlockingRecv error: recv got EOF from 127.28.21.1:57227 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
Aug 06 02:08:22 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:08:22 dist-test-slave-gp6n krb5kdc[4980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.21.2@KRBTEST.COM for oryx/127.28.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:08:22.452682 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:22.447009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45513 (local address 127.28.21.2:39799)
0806 02:08:22.447279 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:22.447304 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:22.447418 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:22.447668 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:22.447670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:22.447887 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:22.447989 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:22.447995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.448745 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.448749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:22.449420 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:22.449430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.449901 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.449904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:22.449973 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:22.450517 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:22.450529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:22.452513 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.21.62:45513: Server oryx/127.28.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
W20260806 02:08:22.452786 6099 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:22.447084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:39799 (local address 127.28.21.62:45513)
0806 02:08:22.447224 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:22.447227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:22.447378 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:08:22.447518 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:22.447520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:22.447563 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:22.447631 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:22.448100 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.448636 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.449555 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.449782 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.449864 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:22.452666 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.21.2:39799: BlockingRecv error: recv got EOF from 127.28.21.2:39799 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:08:22.542537 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5602
I20260806 02:08:22.550794 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5761
I20260806 02:08:22.557190 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5900
I20260806 02:08:22.563098 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 5531
2026-08-06T02:08:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18094 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:22Z Disabled control of system clock
I20260806 02:08:22.595185 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46649
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:08:22.710659 6104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:22.710937 6104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:22.710971 6104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:22.715047 6104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:22.715113 6104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:22.715129 6104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:22.715147 6104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:22.715161 6104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:22.719977 6104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:22.721194 6104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:22.722618 6104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:22.731082 6112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:22.731101 6110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:22.731086 6109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:22.731191 6104 server_base.cc:1034] running on GCE node
I20260806 02:08:22.731894 6104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:22.733013 6104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:22.734205 6104 hybrid_clock.cc:648] HybridClock initialized: now 1785982102734193 us; error 29 us; skew 500 ppm
I20260806 02:08:22.736070 6104 webserver.cc:460] Webserver started at http://127.28.21.62:41077/ using document root <none> and password file <none>
I20260806 02:08:22.736653 6104 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:22.736744 6104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:22.736972 6104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:22.739061 6104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "a22de6b55e014ce5923f712e90a78ead"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "1e2aef39a0254c51f8c7abae8c02ea41"
server_key_iv: "4b07b4bde257c7c1dbeccf7ae4de2b7b"
server_key_version: "encryptionkey@0"
I20260806 02:08:22.739641 6104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "a22de6b55e014ce5923f712e90a78ead"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "1e2aef39a0254c51f8c7abae8c02ea41"
server_key_iv: "4b07b4bde257c7c1dbeccf7ae4de2b7b"
server_key_version: "encryptionkey@0"
I20260806 02:08:22.743288 6104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:08:22.745703 6118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:22.747004 6104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:22.747154 6104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "a22de6b55e014ce5923f712e90a78ead"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "1e2aef39a0254c51f8c7abae8c02ea41"
server_key_iv: "4b07b4bde257c7c1dbeccf7ae4de2b7b"
server_key_version: "encryptionkey@0"
I20260806 02:08:22.747267 6104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:22.771188 6104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:22.771925 6104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:22.772127 6104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:22.779034 6104 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:46649
I20260806 02:08:22.779060 6171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:46649 every 8 connection(s)
I20260806 02:08:22.780009 6104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:22.781152 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6104
I20260806 02:08:22.781273 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:08:22.781577 28756 external_mini_cluster.cc:1428] Setting key 3400c5138a0f667bd2ed8184a628c06b
I20260806 02:08:22.783582 6172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:22.790328 6172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead: Bootstrap starting.
I20260806 02:08:22.792356 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:22.783112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.28.21.62:46649)
0806 02:08:22.783784 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:22.783803 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:22.783846 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:08:22.784691 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:22.784714 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:22.785203 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:22.785548 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:22.786008 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.787514 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.788486 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:22.789185 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:22.789377 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:22.790297 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:22.790303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:22.790320 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:22.790827 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:22.790899 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:22.790926 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:22.791032 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:22.791150 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:08:22.791413 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260806 02:08:22.792955 6172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:22.793797 6172 log.cc:824] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:22.795686 6172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead: No bootstrap required, opened a new log
I20260806 02:08:22.799149 6172 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } }
I20260806 02:08:22.799363 6172 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:22.799448 6172 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22de6b55e014ce5923f712e90a78ead, State: Initialized, Role: FOLLOWER
I20260806 02:08:22.799858 6172 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } }
I20260806 02:08:22.799999 6172 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:22.800064 6172 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:22.800174 6172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:22.801002 6172 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } }
I20260806 02:08:22.801322 6172 leader_election.cc:302] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22de6b55e014ce5923f712e90a78ead; no voters:
I20260806 02:08:22.801616 6172 leader_election.cc:288] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:08:22.801754 6177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:22.801961 6177 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [term 1 LEADER]: Becoming Leader. State: Replica: a22de6b55e014ce5923f712e90a78ead, State: Running, Role: LEADER
I20260806 02:08:22.802301 6177 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } }
I20260806 02:08:22.802842 6172 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:22.804065 6178 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22de6b55e014ce5923f712e90a78ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } } }
I20260806 02:08:22.804105 6179 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22de6b55e014ce5923f712e90a78ead. Latest consensus state: current_term: 1 leader_uuid: "a22de6b55e014ce5923f712e90a78ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22de6b55e014ce5923f712e90a78ead" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46649 } } }
I20260806 02:08:22.804196 6178 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:22.804198 6179 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:22.804571 6186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:22.807157 6186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:22.812629 6186 catalog_manager.cc:1347] Generated new cluster ID: 79b35ed402204be6b7f32b7d5f487e7a
I20260806 02:08:22.812700 6186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:22.829726 6186 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:08:22.830719 6186 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:22.836292 6186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22de6b55e014ce5923f712e90a78ead: Generated new TSK 0
I20260806 02:08:22.836926 6186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:08:22.843566 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--builtin_ntp_servers=127.28.21.20:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:08:22.961833 6196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:22.962121 6196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:22.962191 6196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:22.966274 6196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:22.966369 6196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:22.966488 6196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:22.971344 6196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:22.972492 6196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:22.973932 6196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:22.983629 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:22.983594 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:22.983574 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:22.983664 6196 server_base.cc:1034] running on GCE node
I20260806 02:08:22.984246 6196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:22.984929 6196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:22.986125 6196 hybrid_clock.cc:648] HybridClock initialized: now 1785982102986109 us; error 29 us; skew 500 ppm
I20260806 02:08:22.988095 6196 webserver.cc:460] Webserver started at http://127.28.21.1:35079/ using document root <none> and password file <none>
I20260806 02:08:22.988729 6196 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:22.988828 6196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:22.989081 6196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:22.991477 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "63c68131ef2a4b8ba481eb2b8f44d400"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "3c41fc79bcd4b5c8d8674370e027fbcc"
server_key_iv: "124c128edc1ad3fe00ddc4cee2e469f7"
server_key_version: "encryptionkey@0"
I20260806 02:08:22.992095 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "63c68131ef2a4b8ba481eb2b8f44d400"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "3c41fc79bcd4b5c8d8674370e027fbcc"
server_key_iv: "124c128edc1ad3fe00ddc4cee2e469f7"
server_key_version: "encryptionkey@0"
I20260806 02:08:22.996150 6196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:08:22.999013 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.000329 6196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:08:23.000470 6196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "63c68131ef2a4b8ba481eb2b8f44d400"
format_stamp: "Formatted at 2026-08-06 02:08:22 on dist-test-slave-gp6n"
server_key: "3c41fc79bcd4b5c8d8674370e027fbcc"
server_key_iv: "124c128edc1ad3fe00ddc4cee2e469f7"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.000599 6196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:23.014694 6196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:23.015452 6196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:23.015664 6196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:23.016307 6196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:23.017354 6196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:23.017426 6196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.017504 6196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:23.017555 6196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.027484 6196 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:38809
I20260806 02:08:23.027551 6330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:38809 every 8 connection(s)
I20260806 02:08:23.028455 6196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:23.031606 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6196
I20260806 02:08:23.031723 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:08:23.031978 28756 external_mini_cluster.cc:1428] Setting key 166bd65396fe9fe2f24d695aca0dd1e6
I20260806 02:08:23.034051 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--builtin_ntp_servers=127.28.21.20:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:08:23.038719 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.030438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:46359 (local address 127.28.21.62:46649)
0806 02:08:23.030575 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.030579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.031697 (+ 1118us) server_negotiation.cc:407] Connection header received
0806 02:08:23.033429 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.033433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.033493 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.033600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.034681 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.035266 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.036508 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.036745 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.036860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.038054 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:23.038061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:23.038067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:23.038088 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:23.038115 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:23.038130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:23.038218 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:23.038358 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:08:23.038481 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:08:23.039450 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.030842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46649 (local address 127.28.21.1:46359)
0806 02:08:23.031546 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:23.031589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:23.033186 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:23.033786 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:23.033794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:23.033880 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:23.034538 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.034555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.035401 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.035405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.036367 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.036376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.036902 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.036906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.037090 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.037771 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:23.037773 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:23.037812 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:23.037882 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:23.038223 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:23.038230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:23.038234 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:08:23.038333 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:08:23.038612 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260806 02:08:23.040838 6331 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46649
I20260806 02:08:23.041123 6331 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:23.041692 6331 heartbeater.cc:511] Master 127.28.21.62:46649 requested a full tablet report, sending...
I20260806 02:08:23.043311 6135 ts_manager.cc:194] Registered new tserver with Master: 63c68131ef2a4b8ba481eb2b8f44d400 (127.28.21.1:38809)
I20260806 02:08:23.044611 6135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.1:46359
W20260806 02:08:23.151803 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:23.152107 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:23.152145 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:23.156255 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:23.156325 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:23.156405 6334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:23.161355 6334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:23.162563 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:23.163985 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:23.173602 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.173640 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.173609 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.173826 6334 server_base.cc:1034] running on GCE node
I20260806 02:08:23.174242 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:23.174925 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:23.176136 6334 hybrid_clock.cc:648] HybridClock initialized: now 1785982103176106 us; error 48 us; skew 500 ppm
I20260806 02:08:23.177975 6334 webserver.cc:460] Webserver started at http://127.28.21.2:46431/ using document root <none> and password file <none>
I20260806 02:08:23.178608 6334 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:23.178699 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:23.178926 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:23.180955 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "402ba0e548a54c8ab2e26b332400242b"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "cacf05a6c45b851f6201245df0068887"
server_key_iv: "68c30edc241bd4900330c6e1237f64ee"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.181548 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "402ba0e548a54c8ab2e26b332400242b"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "cacf05a6c45b851f6201245df0068887"
server_key_iv: "68c30edc241bd4900330c6e1237f64ee"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.185148 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:08:23.187619 6348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.188755 6334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:08:23.188896 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "402ba0e548a54c8ab2e26b332400242b"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "cacf05a6c45b851f6201245df0068887"
server_key_iv: "68c30edc241bd4900330c6e1237f64ee"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.189007 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:23.209092 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:23.209774 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:23.209980 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:23.210603 6334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:23.211546 6334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:23.211616 6334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.211683 6334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:23.211730 6334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.220517 6334 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:34909
I20260806 02:08:23.220552 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:34909 every 8 connection(s)
I20260806 02:08:23.221421 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:23.222116 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6334
I20260806 02:08:23.222235 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:08:23.222610 28756 external_mini_cluster.cc:1428] Setting key e0e52f8cee71af35482b0e77da2ca2ad
I20260806 02:08:23.224736 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--builtin_ntp_servers=127.28.21.20:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:08:23.232679 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.223871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:36187 (local address 127.28.21.62:46649)
0806 02:08:23.223996 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.224001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.225248 (+ 1247us) server_negotiation.cc:407] Connection header received
0806 02:08:23.226620 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.226624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.226678 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.226760 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.227872 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.228414 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.230394 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.230711 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.230813 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.232190 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:23.232194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:23.232196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:23.232210 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:23.232230 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:23.232239 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:23.232304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:23.232497 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 02:08:23.232556 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 02:08:23.233975 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.224370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46649 (local address 127.28.21.2:36187)
0806 02:08:23.225082 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:23.225134 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:23.226378 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:23.226934 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:23.226941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:23.227028 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:23.227702 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.227725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.228548 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.228554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.230196 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.230210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.230884 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.230892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.231140 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.231822 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:23.231827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:23.231890 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:23.231983 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:23.232328 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:23.232340 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:23.232349 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:08:23.232493 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:08:23.232847 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260806 02:08:23.235204 6472 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46649
I20260806 02:08:23.235466 6472 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:23.236151 6472 heartbeater.cc:511] Master 127.28.21.62:46649 requested a full tablet report, sending...
I20260806 02:08:23.237304 6135 ts_manager.cc:194] Registered new tserver with Master: 402ba0e548a54c8ab2e26b332400242b (127.28.21.2:34909)
I20260806 02:08:23.237936 6135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.2:36187
W20260806 02:08:23.337862 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:23.338121 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:23.338195 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:23.342203 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:23.342278 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:23.342367 6475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:23.347071 6475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:23.348076 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:23.349423 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:23.358819 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.358850 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.359014 6475 server_base.cc:1034] running on GCE node
W20260806 02:08:23.358839 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.359472 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:23.360081 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:23.361241 6475 hybrid_clock.cc:648] HybridClock initialized: now 1785982103361225 us; error 26 us; skew 500 ppm
I20260806 02:08:23.363026 6475 webserver.cc:460] Webserver started at http://127.28.21.3:36335/ using document root <none> and password file <none>
I20260806 02:08:23.363579 6475 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:23.363641 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:23.363881 6475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:23.365875 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "1fcbd84728c24fb08051684001546772"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "5e6fc391eea3aac90343f6e17d3046f4"
server_key_iv: "1d95576a3eb327d4a27f97f3e755e67b"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.366489 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "1fcbd84728c24fb08051684001546772"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "5e6fc391eea3aac90343f6e17d3046f4"
server_key_iv: "1d95576a3eb327d4a27f97f3e755e67b"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.370042 6475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:08:23.372470 6489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.373612 6475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:23.373703 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "1fcbd84728c24fb08051684001546772"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "5e6fc391eea3aac90343f6e17d3046f4"
server_key_iv: "1d95576a3eb327d4a27f97f3e755e67b"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.373782 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:23.385963 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:23.386600 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:23.386757 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:23.387277 6475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:23.388115 6475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:23.388163 6475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.388201 6475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:23.388217 6475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.397217 6475 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:33613
I20260806 02:08:23.397238 6609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:33613 every 8 connection(s)
I20260806 02:08:23.398278 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:08:23.402611 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6475
I20260806 02:08:23.402722 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:08:23.403012 28756 external_mini_cluster.cc:1428] Setting key 7445e9bbc48980e32969dccb571a6cde
I20260806 02:08:23.408548 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.400219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:46211 (local address 127.28.21.62:46649)
0806 02:08:23.400387 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.400391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.401443 (+ 1052us) server_negotiation.cc:407] Connection header received
0806 02:08:23.403135 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.403139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.403200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.403287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.404437 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.405168 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.406448 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.406710 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.406829 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.407953 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:23.407957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:23.407960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:23.407973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:23.407998 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:23.408014 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:23.408096 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:23.408267 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:08:23.408384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:08:23.409392 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.400607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46649 (local address 127.28.21.3:46211)
0806 02:08:23.401281 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:23.401334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:23.402919 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:23.403519 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:23.403529 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:23.403667 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:23.404291 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.404310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.405295 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.405299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.406303 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:23.406314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:23.406875 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:23.406879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:23.407066 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:23.407663 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:23.407664 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:23.407707 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:23.407774 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:23.408099 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:23.408112 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:23.408120 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:08:23.408254 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:08:23.408553 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260806 02:08:23.410497 6610 heartbeater.cc:347] Connected to a master server at 127.28.21.62:46649
I20260806 02:08:23.410741 6610 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:23.411274 6610 heartbeater.cc:511] Master 127.28.21.62:46649 requested a full tablet report, sending...
I20260806 02:08:23.412276 6135 ts_manager.cc:194] Registered new tserver with Master: 1fcbd84728c24fb08051684001546772 (127.28.21.3:33613)
I20260806 02:08:23.412871 6135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.3:46211
I20260806 02:08:23.417171 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:08:23.420056 6175 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:23.419212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.28.21.62:46649)
0806 02:08:23.419335 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.419340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.419486 (+ 146us) server_negotiation.cc:407] Connection header received
0806 02:08:23.419584 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.419587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.419626 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.419699 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.419912 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51124: BlockingRecv error: recv got EOF from 127.0.0.1:51124 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:08:23.420717 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6196
I20260806 02:08:23.428244 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6334
I20260806 02:08:23.434859 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6475
I20260806 02:08:23.440940 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6104
2026-08-06T02:08:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (877 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:23Z Disabled control of system clock
I20260806 02:08:23.472859 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37495
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:37495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:08:23.582736 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:23.582983 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:23.583019 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:23.586968 6621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:23.587018 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:23.587033 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:23.587078 6621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:23.587109 6621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:23.591776 6621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:37495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:23.592734 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:23.594170 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:23.602540 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.602638 6621 server_base.cc:1034] running on GCE node
W20260806 02:08:23.602579 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.602596 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.603322 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:23.604362 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:23.605545 6621 hybrid_clock.cc:648] HybridClock initialized: now 1785982103605534 us; error 25 us; skew 500 ppm
I20260806 02:08:23.607358 6621 webserver.cc:460] Webserver started at http://127.28.21.62:44655/ using document root <none> and password file <none>
I20260806 02:08:23.607897 6621 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:23.607985 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:23.608209 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:23.610145 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "221111005f5e4d2cb764579e584fe099"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "8b470715ca913fa5a6f3cca74607465d"
server_key_iv: "5d6a9aae20441e432298d3f798b7c0c5"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.610704 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "221111005f5e4d2cb764579e584fe099"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "8b470715ca913fa5a6f3cca74607465d"
server_key_iv: "5d6a9aae20441e432298d3f798b7c0c5"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.614106 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:08:23.616441 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.617599 6621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:23.617736 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "221111005f5e4d2cb764579e584fe099"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "8b470715ca913fa5a6f3cca74607465d"
server_key_iv: "5d6a9aae20441e432298d3f798b7c0c5"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.617841 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:23.622210 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:23.622859 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:23.623036 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:23.629490 6621 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:37495
I20260806 02:08:23.629544 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:37495 every 8 connection(s)
I20260806 02:08:23.630391 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:23.633618 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:23.638518 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6621
I20260806 02:08:23.638639 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:08:23.638952 28756 external_mini_cluster.cc:1428] Setting key a16d2d3fe0bb158f8cd9e68d6c2d6c77
I20260806 02:08:23.639168 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099: Bootstrap starting.
I20260806 02:08:23.641688 6690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:23.642470 6690 log.cc:824] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:23.644304 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.640312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.28.21.62:37495)
0806 02:08:23.640943 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.640957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.640990 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:08:23.641611 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.641627 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.642001 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.642315 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.643137 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:23.643142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:23.643154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:23.643164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:23.643216 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:23.643232 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:23.643341 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:23.643441 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 02:08:23.643681 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260806 02:08:23.644371 6690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099: No bootstrap required, opened a new log
I20260806 02:08:23.648195 6690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } }
I20260806 02:08:23.648411 6690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:23.648494 6690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221111005f5e4d2cb764579e584fe099, State: Initialized, Role: FOLLOWER
I20260806 02:08:23.648874 6690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } }
I20260806 02:08:23.649009 6690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:23.649070 6690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:23.649183 6690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:23.650084 6690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } }
I20260806 02:08:23.650446 6690 leader_election.cc:302] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221111005f5e4d2cb764579e584fe099; no voters:
I20260806 02:08:23.650767 6690 leader_election.cc:288] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:08:23.650841 6695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:23.651065 6695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [term 1 LEADER]: Becoming Leader. State: Replica: 221111005f5e4d2cb764579e584fe099, State: Running, Role: LEADER
I20260806 02:08:23.651358 6695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } }
I20260806 02:08:23.651993 6690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:23.652448 6696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221111005f5e4d2cb764579e584fe099. Latest consensus state: current_term: 1 leader_uuid: "221111005f5e4d2cb764579e584fe099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } } }
I20260806 02:08:23.652573 6696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:23.652874 6697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221111005f5e4d2cb764579e584fe099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221111005f5e4d2cb764579e584fe099" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37495 } } }
I20260806 02:08:23.652976 6697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:23.653330 6699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:23.656286 6699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:23.661618 6699 catalog_manager.cc:1347] Generated new cluster ID: 1bdfcca3237d438a9c2b9bfe5347ce09
I20260806 02:08:23.661689 6699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:23.668921 6699 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:08:23.669252 6699 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:23.678449 6699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221111005f5e4d2cb764579e584fe099: Generated new TSK 0
I20260806 02:08:23.679050 6699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:08:23.687551 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--builtin_ntp_servers=127.28.21.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:08:23.816668 6715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:23.816911 6715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:23.816951 6715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:23.820945 6715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:23.821000 6715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:23.821084 6715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:23.825784 6715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:23.826894 6715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:23.828289 6715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:08:23.838258 6715 server_base.cc:1034] running on GCE node
W20260806 02:08:23.838279 6720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.838232 6723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:23.838266 6721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:23.838963 6715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:23.839623 6715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:23.840785 6715 hybrid_clock.cc:648] HybridClock initialized: now 1785982103840766 us; error 32 us; skew 500 ppm
I20260806 02:08:23.842707 6715 webserver.cc:460] Webserver started at http://127.28.21.1:40659/ using document root <none> and password file <none>
I20260806 02:08:23.843309 6715 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:23.843371 6715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:23.843582 6715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:23.845592 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "852b62af59f94222bc66496d14335a9e"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "480fa0fb2f6d4ba9ef9df6caaf50a891"
server_key_iv: "6d2e6f7aedb3af5aaf0a3ceea55df38d"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.846148 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "852b62af59f94222bc66496d14335a9e"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "480fa0fb2f6d4ba9ef9df6caaf50a891"
server_key_iv: "6d2e6f7aedb3af5aaf0a3ceea55df38d"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.849714 6715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:08:23.852303 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.853634 6715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:23.853760 6715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "852b62af59f94222bc66496d14335a9e"
format_stamp: "Formatted at 2026-08-06 02:08:23 on dist-test-slave-gp6n"
server_key: "480fa0fb2f6d4ba9ef9df6caaf50a891"
server_key_iv: "6d2e6f7aedb3af5aaf0a3ceea55df38d"
server_key_version: "encryptionkey@0"
I20260806 02:08:23.853868 6715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:23.858249 6715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:23.858887 6715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:23.859073 6715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:23.859591 6715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:23.860450 6715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:23.860519 6715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.860589 6715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:23.860630 6715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:23.869769 6715 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:44617
I20260806 02:08:23.869797 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:44617 every 8 connection(s)
I20260806 02:08:23.870718 6715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:23.875130 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6715
I20260806 02:08:23.875252 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:08:23.875623 28756 external_mini_cluster.cc:1428] Setting key 62258ad105476183c5b7dce0857a82bb
I20260806 02:08:23.876684 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.872355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:35285 (local address 127.28.21.62:37495)
0806 02:08:23.872557 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:23.872562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:23.873542 (+ 980us) server_negotiation.cc:407] Connection header received
0806 02:08:23.874860 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:23.874863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:23.874898 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:23.874962 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:23.876187 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:23.876193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:23.876198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:23.876213 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:23.876236 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:23.876251 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:23.876322 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:23.876487 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:08:23.876557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:08:23.877710 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:23.872743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37495 (local address 127.28.21.1:35285)
0806 02:08:23.873386 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:23.873434 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:23.874661 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:23.875130 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:23.875139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:23.875257 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:23.875887 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:23.875890 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:23.875942 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:23.876029 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:23.876333 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:23.876342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:23.876348 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:08:23.876534 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 02:08:23.876755 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260806 02:08:23.877859 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--builtin_ntp_servers=127.28.21.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:08:23.878945 6850 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37495
I20260806 02:08:23.879164 6850 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:23.879550 6850 heartbeater.cc:511] Master 127.28.21.62:37495 requested a full tablet report, sending...
I20260806 02:08:23.880965 6652 ts_manager.cc:194] Registered new tserver with Master: 852b62af59f94222bc66496d14335a9e (127.28.21.1:44617)
W20260806 02:08:23.989172 6853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:23.989449 6853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:23.989517 6853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:23.993462 6853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:23.993512 6853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:23.993609 6853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:23.998566 6853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:23.999625 6853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:24.001005 6853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:24.010259 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:24.010234 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:24.010334 6861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:24.010528 6853 server_base.cc:1034] running on GCE node
I20260806 02:08:24.010985 6853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:24.011641 6853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:24.012820 6853 hybrid_clock.cc:648] HybridClock initialized: now 1785982104012798 us; error 34 us; skew 500 ppm
I20260806 02:08:24.014739 6853 webserver.cc:460] Webserver started at http://127.28.21.2:38741/ using document root <none> and password file <none>
I20260806 02:08:24.015309 6853 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:24.015398 6853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:24.015621 6853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:24.017606 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "43529898813844b7a3c253b19f04bfc6"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "28d25d5c9e203a2857320ff7e1a8231b"
server_key_iv: "e7b52569061cd65a81c71725e14e1366"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.018159 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "43529898813844b7a3c253b19f04bfc6"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "28d25d5c9e203a2857320ff7e1a8231b"
server_key_iv: "e7b52569061cd65a81c71725e14e1366"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.021719 6853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:08:24.024554 6867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.025728 6853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:24.025885 6853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "43529898813844b7a3c253b19f04bfc6"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "28d25d5c9e203a2857320ff7e1a8231b"
server_key_iv: "e7b52569061cd65a81c71725e14e1366"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.026001 6853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:24.030663 6853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:24.031253 6853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:24.031436 6853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:24.031965 6853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:24.032850 6853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:24.032917 6853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.032979 6853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:24.033023 6853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.041661 6853 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:42743
I20260806 02:08:24.041693 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:42743 every 8 connection(s)
I20260806 02:08:24.042572 6853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:24.044407 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6853
I20260806 02:08:24.044540 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:08:24.044857 28756 external_mini_cluster.cc:1428] Setting key 02f87776b40a10027d1825ddcb820931
I20260806 02:08:24.046772 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--builtin_ntp_servers=127.28.21.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:08:24.048843 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:24.044311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:54871 (local address 127.28.21.62:37495)
0806 02:08:24.044478 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:24.044484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:24.045659 (+ 1175us) server_negotiation.cc:407] Connection header received
0806 02:08:24.047160 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:24.047164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:24.047206 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:24.047293 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:24.048412 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:24.048416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:24.048419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:24.048432 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:24.048460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:24.048468 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:24.048544 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:24.048704 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:08:24.048741 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:08:24.050101 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:24.044788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37495 (local address 127.28.21.2:54871)
0806 02:08:24.045511 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:24.045561 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:24.046936 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:24.047441 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:24.047448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:24.047533 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:24.048100 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:24.048102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:24.048141 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:24.048230 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:24.048556 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:24.048565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:24.048573 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:08:24.048748 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 02:08:24.049006 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
I20260806 02:08:24.051554 6992 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37495
I20260806 02:08:24.051772 6992 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:24.052203 6992 heartbeater.cc:511] Master 127.28.21.62:37495 requested a full tablet report, sending...
I20260806 02:08:24.053356 6652 ts_manager.cc:194] Registered new tserver with Master: 43529898813844b7a3c253b19f04bfc6 (127.28.21.2:42743)
W20260806 02:08:24.172045 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:24.172456 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:24.172529 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:24.178634 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:24.178697 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:24.178785 6995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:24.184458 6995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:37495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:24.185741 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:24.187225 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:24.196688 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:24.196695 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:24.196695 7003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:24.196851 6995 server_base.cc:1034] running on GCE node
I20260806 02:08:24.197266 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:24.197888 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:24.199040 6995 hybrid_clock.cc:648] HybridClock initialized: now 1785982104199025 us; error 26 us; skew 500 ppm
I20260806 02:08:24.200872 6995 webserver.cc:460] Webserver started at http://127.28.21.3:35487/ using document root <none> and password file <none>
I20260806 02:08:24.201426 6995 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:24.201486 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:24.201706 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:24.203728 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "b1a7af14acbd43aeb448983ef5e49e5c"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "0262cbf7be10242d2fe700cb7ee4844f"
server_key_iv: "e64509edd3290d7c8f441923ffc2f04b"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.204291 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "b1a7af14acbd43aeb448983ef5e49e5c"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "0262cbf7be10242d2fe700cb7ee4844f"
server_key_iv: "e64509edd3290d7c8f441923ffc2f04b"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.207929 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:08:24.210397 7009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.211594 6995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:08:24.211721 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "b1a7af14acbd43aeb448983ef5e49e5c"
format_stamp: "Formatted at 2026-08-06 02:08:24 on dist-test-slave-gp6n"
server_key: "0262cbf7be10242d2fe700cb7ee4844f"
server_key_iv: "e64509edd3290d7c8f441923ffc2f04b"
server_key_version: "encryptionkey@0"
I20260806 02:08:24.211823 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:24.216312 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:24.217074 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:24.217263 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:24.217962 6995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:24.219213 6995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:24.219285 6995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.219386 6995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:24.219429 6995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:24.229974 6995 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:44729
I20260806 02:08:24.230037 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:44729 every 8 connection(s)
I20260806 02:08:24.230959 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:08:24.234532 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 6995
I20260806 02:08:24.234656 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:08:24.235006 28756 external_mini_cluster.cc:1428] Setting key 2848e1dd943a0e0705cd2ae154ceae65
I20260806 02:08:24.237640 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:24.232712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:36263 (local address 127.28.21.62:37495)
0806 02:08:24.232846 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:24.232850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:24.233929 (+ 1079us) server_negotiation.cc:407] Connection header received
0806 02:08:24.235737 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:24.235741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:24.235786 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:24.235879 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:24.237188 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:24.237192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:24.237195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:08:24.237208 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:24.237231 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:08:24.237240 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:24.237331 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:24.237506 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 02:08:24.237544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:08:24.238544 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:24.233128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:37495 (local address 127.28.21.3:36263)
0806 02:08:24.233779 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:24.233817 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:24.235479 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:24.236099 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:24.236108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:24.236220 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:24.236917 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:08:24.236918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:24.236958 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:08:24.237025 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:24.237335 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:24.237346 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:24.237352 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:08:24.237528 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 02:08:24.237739 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260806 02:08:24.239557 7128 heartbeater.cc:347] Connected to a master server at 127.28.21.62:37495
I20260806 02:08:24.239766 7128 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:24.240154 7128 heartbeater.cc:511] Master 127.28.21.62:37495 requested a full tablet report, sending...
I20260806 02:08:24.241067 6652 ts_manager.cc:194] Registered new tserver with Master: b1a7af14acbd43aeb448983ef5e49e5c (127.28.21.3:44729)
I20260806 02:08:24.248724 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:08:24.251463 6693 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:08:24.250609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.28.21.62:37495)
0806 02:08:24.250741 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:24.250746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:24.250904 (+ 158us) server_negotiation.cc:407] Connection header received
0806 02:08:24.251002 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:24.251004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:24.251032 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:24.251095 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:24.251320 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33996: BlockingRecv error: recv got EOF from 127.0.0.1:33996 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:08:24.251623 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6715
I20260806 02:08:24.258206 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6853
I20260806 02:08:24.264102 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6995
I20260806 02:08:24.270929 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 6621
2026-08-06T02:08:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (829 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:08:24 dist-test-slave-gp6n krb5kdc[7138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:08:24 dist-test-slave-gp6n krb5kdc[7138](info): set up 2 sockets
Aug 06 02:08:24 dist-test-slave-gp6n krb5kdc[7138](info): commencing operation
krb5kdc: starting...
W20260806 02:08:25.403334 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:08:25 dist-test-slave-gp6n krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:08:25.530227 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42495
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:25.641710 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:25.642006 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:25.642077 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:25.646080 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:08:25.646165 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:25.646204 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:25.646258 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:08:25.646294 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:08:25.651547 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:42495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:25.652593 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:25.653625 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:25.662220 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:25.662226 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:25.662245 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:25.662650 7151 server_base.cc:1034] running on GCE node
I20260806 02:08:25.663231 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:25.664343 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:25.665508 7151 hybrid_clock.cc:648] HybridClock initialized: now 1785982105665493 us; error 28 us; skew 500 ppm
Aug 06 02:08:25 dist-test-slave-gp6n krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:25.672206 7151 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:08:25.673282 7151 webserver.cc:460] Webserver started at http://127.28.21.62:37719/ using document root <none> and password file <none>
I20260806 02:08:25.673874 7151 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:25.673944 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:25.674206 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:25.676421 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "165103ac6c694048a0175d516c849a33"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "995be5b8f85dc730b52a269ac1239e61"
server_key_iv: "2c90b18cb9d3d58832d759bc899101b8"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.677000 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "165103ac6c694048a0175d516c849a33"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "995be5b8f85dc730b52a269ac1239e61"
server_key_iv: "2c90b18cb9d3d58832d759bc899101b8"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.680814 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:08:25.683296 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:25.684564 7151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:25.684695 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "165103ac6c694048a0175d516c849a33"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "995be5b8f85dc730b52a269ac1239e61"
server_key_iv: "2c90b18cb9d3d58832d759bc899101b8"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.684800 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:25.706583 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:25.709765 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:25.710074 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:25.717367 7151 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:42495
I20260806 02:08:25.717459 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:42495 every 8 connection(s)
I20260806 02:08:25.718338 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:08:25.721282 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:25.727366 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 7151
I20260806 02:08:25.727494 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:08:25.727495 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: Bootstrap starting.
I20260806 02:08:25.727837 28756 external_mini_cluster.cc:1428] Setting key b371cf92d277ed1a9f000cb0eb09b44b
I20260806 02:08:25.730340 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:25.731190 7224 log.cc:824] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:25.733211 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: No bootstrap required, opened a new log
Aug 06 02:08:25 dist-test-slave-gp6n krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:08:25.737537 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } }
I20260806 02:08:25.737816 7224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:25.737892 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165103ac6c694048a0175d516c849a33, State: Initialized, Role: FOLLOWER
I20260806 02:08:25.738476 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } }
I20260806 02:08:25.738662 7224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:08:25.738765 7224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:08:25.738884 7224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:25.739820 7224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } }
I20260806 02:08:25.740175 7224 leader_election.cc:302] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165103ac6c694048a0175d516c849a33; no voters:
I20260806 02:08:25.740492 7224 leader_election.cc:288] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:08:25.740617 7229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:25.740906 7229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [term 1 LEADER]: Becoming Leader. State: Replica: 165103ac6c694048a0175d516c849a33, State: Running, Role: LEADER
I20260806 02:08:25.741277 7229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } }
I20260806 02:08:25.741942 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:08:25.742758 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165103ac6c694048a0175d516c849a33. Latest consensus state: current_term: 1 leader_uuid: "165103ac6c694048a0175d516c849a33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } } }
I20260806 02:08:25.743008 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:25.743244 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:25.729384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.28.21.62:42495)
0806 02:08:25.729845 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:25.729860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:25.729895 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:08:25.730608 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:25.730626 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:25.730953 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:25.731275 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:25.732098 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:25.733150 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:25.734024 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:25.734542 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:25.734705 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:25.737441 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:25.737446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:25.737459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:25.737485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:25.739568 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:25.740101 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:25.740108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:25.740114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:25.740245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:25.740586 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:25.740591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:25.740596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:25.741190 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:25.741427 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:25.741740 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 02:08:25.742065 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260806 02:08:25.743249 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165103ac6c694048a0175d516c849a33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165103ac6c694048a0175d516c849a33" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42495 } } }
I20260806 02:08:25.743552 7239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:08:25.743639 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33 [sys.catalog]: This master's current role is: LEADER
I20260806 02:08:25.746994 7239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:08:25.752812 7239 catalog_manager.cc:1347] Generated new cluster ID: 1f3aad86de7e40d390e0613b8a59ed62
I20260806 02:08:25.752892 7239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:08:25.784180 7239 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:08:25.785579 7239 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:08:25.799294 7239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: Generated new TSK 0
I20260806 02:08:25.800271 7239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:08:25.842747 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:25.955879 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:25.956177 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:25.956251 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:25.960227 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:25.960309 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:25.960395 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:08:25.965734 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:25.966922 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:25.968043 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:25.977826 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:25.977819 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:25.977826 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:25.978070 7252 server_base.cc:1034] running on GCE node
I20260806 02:08:25.978650 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:25.979383 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:25.980588 7252 hybrid_clock.cc:648] HybridClock initialized: now 1785982105980574 us; error 34 us; skew 500 ppm
Aug 06 02:08:25 dist-test-slave-gp6n krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:25.983563 7252 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:08:25.984517 7252 webserver.cc:460] Webserver started at http://127.28.21.1:46285/ using document root <none> and password file <none>
I20260806 02:08:25.985126 7252 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:25.985220 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:25.985468 7252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:25.987730 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "6bb13f63d46b4084838fe7c7c714a2db"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "ab1af8671e7e9fd3e8583271dfbb9d57"
server_key_iv: "8114f44e96546ab8c7c4e0f6e97e0159"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.988363 7252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "6bb13f63d46b4084838fe7c7c714a2db"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "ab1af8671e7e9fd3e8583271dfbb9d57"
server_key_iv: "8114f44e96546ab8c7c4e0f6e97e0159"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.992569 7252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:08:25.995311 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:25.996716 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:08:25.996829 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "6bb13f63d46b4084838fe7c7c714a2db"
format_stamp: "Formatted at 2026-08-06 02:08:25 on dist-test-slave-gp6n"
server_key: "ab1af8671e7e9fd3e8583271dfbb9d57"
server_key_iv: "8114f44e96546ab8c7c4e0f6e97e0159"
server_key_version: "encryptionkey@0"
I20260806 02:08:25.996932 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:26.013593 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:26.016979 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:26.017150 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:26.017724 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:26.018604 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:26.018656 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.018697 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:26.018712 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.027853 7252 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:36247
I20260806 02:08:26.027891 7386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:36247 every 8 connection(s)
I20260806 02:08:26.028848 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:08:26.030149 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 7252
I20260806 02:08:26.030272 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:08:26.030611 28756 external_mini_cluster.cc:1428] Setting key 8130d24d3454b5f9c272185bf591b77d
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:08:26.044512 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.031061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:60473 (local address 127.28.21.62:42495)
0806 02:08:26.031223 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.031228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.032131 (+ 903us) server_negotiation.cc:407] Connection header received
0806 02:08:26.033621 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.033624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.033677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.033780 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:26.035087 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.035953 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.037249 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.037710 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.037845 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.040407 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:26.040412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:26.040418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:26.040452 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:26.042034 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.042733 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.042735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.042737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.042815 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.043222 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.043224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.043225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.043470 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:26.043603 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.044251 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.044364 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:08:26.045905 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.031491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42495 (local address 127.28.21.1:60473)
0806 02:08:26.031972 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.032038 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.033404 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.033961 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.033968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.034339 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:26.034936 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.034950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.036087 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.036090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.037119 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.037129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.037877 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.037880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.038023 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.038728 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:26.038745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:26.040222 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:26.042167 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.042172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.042213 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.042641 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.042901 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.042904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.042905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.043095 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.043623 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:26.043630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:26.043983 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.044184 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.044619 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260806 02:08:26.047294 7387 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42495
I20260806 02:08:26.047600 7387 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:26.048185 7387 heartbeater.cc:511] Master 127.28.21.62:42495 requested a full tablet report, sending...
I20260806 02:08:26.049809 7186 ts_manager.cc:194] Registered new tserver with Master: 6bb13f63d46b4084838fe7c7c714a2db (127.28.21.1:36247)
I20260806 02:08:26.051113 7186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:60473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:08:26.062764 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:26.177168 7392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:26.177428 7392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:26.177472 7392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:26.181598 7392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:26.181658 7392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:26.181744 7392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:08:26.187155 7392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:26.188143 7392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:26.189193 7392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:08:26.198541 7399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:26.198550 7401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:26.198587 7398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:26.198930 7392 server_base.cc:1034] running on GCE node
I20260806 02:08:26.199301 7392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:26.199987 7392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:26.201196 7392 hybrid_clock.cc:648] HybridClock initialized: now 1785982106201165 us; error 36 us; skew 500 ppm
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:26.204205 7392 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:08:26.205133 7392 webserver.cc:460] Webserver started at http://127.28.21.2:46189/ using document root <none> and password file <none>
I20260806 02:08:26.205719 7392 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:26.205787 7392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:26.206048 7392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:26.208300 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "47281c5bd05ff63efad6d77ad5b69aed"
server_key_iv: "ebb941242a69233d99cd9dbacbbb9974"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.208909 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "47281c5bd05ff63efad6d77ad5b69aed"
server_key_iv: "ebb941242a69233d99cd9dbacbbb9974"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.212946 7392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:08:26.215458 7408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.216747 7392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:26.216902 7392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "47281c5bd05ff63efad6d77ad5b69aed"
server_key_iv: "ebb941242a69233d99cd9dbacbbb9974"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.217015 7392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:26.232934 7392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:26.235998 7392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:26.236209 7392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:26.236841 7392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:26.237717 7392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:26.237792 7392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.237895 7392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:26.237944 7392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.248126 7392 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:34467
I20260806 02:08:26.248145 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:34467 every 8 connection(s)
I20260806 02:08:26.249071 7392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:08:26.250013 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 7392
I20260806 02:08:26.250178 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:08:26.250554 28756 external_mini_cluster.cc:1428] Setting key 6d023671fa75dc14d0fcfd50ff9cb0c7
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:08:26.265290 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.250984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:34935 (local address 127.28.21.62:42495)
0806 02:08:26.251137 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.251142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.252096 (+ 954us) server_negotiation.cc:407] Connection header received
0806 02:08:26.254331 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.254334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.254375 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.254490 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:26.255842 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.256699 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.257981 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.258233 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.258294 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.261517 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:26.261521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:26.261525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:26.261557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:26.263384 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.263875 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.263878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.263879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.263964 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.264266 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.264271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.264276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.264501 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:26.264587 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.265031 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.265160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:08:26.266093 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.251406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42495 (local address 127.28.21.2:34935)
0806 02:08:26.251910 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.251991 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.254093 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.254682 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.254692 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.255066 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:26.255687 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.255709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.256827 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.256832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.257798 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.257808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.258544 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.258551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.258801 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.259515 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:26.259545 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:26.261346 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:26.263523 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.263528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.263540 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.263783 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.264045 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.264047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.264048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.264173 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.264599 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:26.264604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:26.264825 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.264978 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.265232 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:08:26.267356 7530 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42495
I20260806 02:08:26.267620 7530 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:26.268131 7530 heartbeater.cc:511] Master 127.28.21.62:42495 requested a full tablet report, sending...
I20260806 02:08:26.269263 7186 ts_manager.cc:194] Registered new tserver with Master: 67e9e93e740b4ba8bbbb62f78f6f7f32 (127.28.21.2:34467)
I20260806 02:08:26.269887 7186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:34935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:08:26.283571 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:08:26.397504 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:08:26.397754 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:08:26.397831 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:08:26.401969 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:08:26.402029 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:08:26.402127 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:08:26.407361 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:08:26.408380 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:08:26.409395 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:08:26.418792 7535 server_base.cc:1034] running on GCE node
W20260806 02:08:26.418746 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:26.418741 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:08:26.418742 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:08:26.419437 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:08:26.420038 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:08:26.421195 7535 hybrid_clock.cc:648] HybridClock initialized: now 1785982106421162 us; error 43 us; skew 500 ppm
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:08:26.423982 7535 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:08:26.424760 7535 webserver.cc:460] Webserver started at http://127.28.21.3:39763/ using document root <none> and password file <none>
I20260806 02:08:26.425278 7535 fs_manager.cc:359] Metadata directory not provided
I20260806 02:08:26.425338 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:08:26.425503 7535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:08:26.427526 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "616fb3f80d684867906975e6fd7066e4"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "535cd596e6459141ea229f484370dbad"
server_key_iv: "5e8529e91ac5f278c6ef4caf619ccd3b"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.428043 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "616fb3f80d684867906975e6fd7066e4"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "535cd596e6459141ea229f484370dbad"
server_key_iv: "5e8529e91ac5f278c6ef4caf619ccd3b"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.431834 7535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:08:26.434128 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.435432 7535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:08:26.435549 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "616fb3f80d684867906975e6fd7066e4"
format_stamp: "Formatted at 2026-08-06 02:08:26 on dist-test-slave-gp6n"
server_key: "535cd596e6459141ea229f484370dbad"
server_key_iv: "5e8529e91ac5f278c6ef4caf619ccd3b"
server_key_version: "encryptionkey@0"
I20260806 02:08:26.435645 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:08:26.450572 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:08:26.453606 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:08:26.453806 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:08:26.454382 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:08:26.455240 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:08:26.455286 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.455345 7535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:08:26.455377 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:08:26.464378 7535 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:42543
I20260806 02:08:26.464440 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:42543 every 8 connection(s)
I20260806 02:08:26.465306 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:08:26.469949 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 7535
I20260806 02:08:26.470088 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:08:26.470361 28756 external_mini_cluster.cc:1428] Setting key 7976ffbccc6fbb6bc008b562695af187
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:08:26.479278 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.467084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:51911 (local address 127.28.21.62:42495)
0806 02:08:26.467228 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.467231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.468059 (+ 828us) server_negotiation.cc:407] Connection header received
0806 02:08:26.469472 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.469476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.469536 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.469641 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:26.471076 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.471804 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.473021 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.473255 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.473381 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.475910 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:26.475914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:26.475917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:26.475944 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:26.477192 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.477703 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.477706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.477708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.477796 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.478152 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.478155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.478157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.478345 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:26.478472 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.478986 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.479112 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:08:26.480089 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.467454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42495 (local address 127.28.21.3:51911)
0806 02:08:26.467909 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.467951 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.469244 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.469815 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.469822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.470274 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:08:26.470933 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.470951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.471925 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.471929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.472898 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.472908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.473424 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.473427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.473595 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.474141 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:08:26.474159 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:08:26.475735 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:08:26.477307 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.477313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.477329 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.477595 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.477898 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:08:26.477901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:08:26.477903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:08:26.478048 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:08:26.478471 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:08:26.478478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:08:26.478725 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.478944 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.479272 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260806 02:08:26.481225 7673 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42495
I20260806 02:08:26.481483 7673 heartbeater.cc:464] Registering TS with master...
I20260806 02:08:26.481984 7673 heartbeater.cc:511] Master 127.28.21.62:42495 requested a full tablet report, sending...
I20260806 02:08:26.483163 7186 ts_manager.cc:194] Registered new tserver with Master: 616fb3f80d684867906975e6fd7066e4 (127.28.21.3:42543)
I20260806 02:08:26.483904 7186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:51911
I20260806 02:08:26.484217 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:08:26.494273 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.486533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.28.21.62:42495)
0806 02:08:26.486658 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.486663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.486883 (+ 220us) server_negotiation.cc:407] Connection header received
0806 02:08:26.487043 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.487046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.487091 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.487185 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:08:26.487958 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.488573 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.489599 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.489854 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.489969 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.491400 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:08:26.491408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:08:26.491411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:08:26.491434 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:08:26.492616 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.493048 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.493051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.493053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.493138 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:08:26.493403 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:08:26.493407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:08:26.493409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:08:26.493629 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:08:26.493766 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.493997 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.494115 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:08:26.496887 7186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:08:26.499981 7186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:08:26.514442 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.509150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:34467 (local address 127.0.0.1:57184)
0806 02:08:26.509607 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.509620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.509699 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.510446 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.510448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.510471 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:26.510641 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.510649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.511637 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.511643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.513072 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.513083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.513869 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.513872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.513985 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.514034 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.514140 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.514254 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":59,"thread_start_us":92,"threads_started":1}
I20260806 02:08:26.514492 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.508617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36247 (local address 127.0.0.1:58378)
0806 02:08:26.509243 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.509259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.509415 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.510378 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.510381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.510397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:26.510658 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.510664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.512028 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.512036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.513111 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.513120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.514148 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.514151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.514230 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.514248 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.514304 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.514358 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":176,"thread_start_us":114,"threads_started":1}
I20260806 02:08:26.514966 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.509668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.28.21.2:34467)
0806 02:08:26.509996 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.510008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.510029 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:08:26.510099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.510104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.510292 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.510466 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:26.510776 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.511515 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.513192 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.513750 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.513870 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.514021 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.514774 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.514855 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":153,"threads_started":1}
I20260806 02:08:26.515010 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.509207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.28.21.1:36247)
0806 02:08:26.509753 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.509769 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.509791 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:08:26.509863 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.509870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.510056 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.510207 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:26.510762 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.511879 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.513230 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.514039 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.514150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.514282 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.514338 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.514476 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":111,"threads_started":1}
I20260806 02:08:26.515002 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.508430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42543 (local address 127.0.0.1:44794)
0806 02:08:26.508964 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.509013 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.509138 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.509984 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.509989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.510012 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:26.510171 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.510178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.511975 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.511978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.513396 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.513405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.514492 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.514495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.514736 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.514757 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.514801 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.514863 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260806 02:08:26.515847 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.508573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.28.21.3:42543)
0806 02:08:26.509447 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.509463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.509489 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:08:26.509584 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.509592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.509809 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.510314 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:26.510614 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.511835 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.513530 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.514331 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.515272 (+ 941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.515420 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.515465 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.515543 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":152,"threads_started":1}
I20260806 02:08:26.517709 7319 tablet_service.cc:1467] Processing CreateTablet for tablet eef15dca467c4c23915a8757ab526ac1 (DEFAULT_TABLE table=test-table [id=fea344238fa14b01a95c60481e75a91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:26.517793 7459 tablet_service.cc:1467] Processing CreateTablet for tablet eef15dca467c4c23915a8757ab526ac1 (DEFAULT_TABLE table=test-table [id=fea344238fa14b01a95c60481e75a91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:26.518268 7601 tablet_service.cc:1467] Processing CreateTablet for tablet eef15dca467c4c23915a8757ab526ac1 (DEFAULT_TABLE table=test-table [id=fea344238fa14b01a95c60481e75a91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:08:26.518781 7319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef15dca467c4c23915a8757ab526ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:26.518857 7459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef15dca467c4c23915a8757ab526ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:26.519184 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef15dca467c4c23915a8757ab526ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:08:26.525661 7691 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: Bootstrap starting.
I20260806 02:08:26.526368 7692 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Bootstrap starting.
I20260806 02:08:26.526436 7693 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Bootstrap starting.
I20260806 02:08:26.528205 7691 tablet_bootstrap.cc:654] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:26.528460 7692 tablet_bootstrap.cc:654] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:26.528671 7693 tablet_bootstrap.cc:654] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:08:26.529299 7692 log.cc:824] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:26.529518 7691 log.cc:824] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:26.529518 7693 log.cc:824] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:08:26.531440 7691 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: No bootstrap required, opened a new log
I20260806 02:08:26.531441 7693 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: No bootstrap required, opened a new log
I20260806 02:08:26.531440 7692 tablet_bootstrap.cc:492] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: No bootstrap required, opened a new log
I20260806 02:08:26.531644 7691 ts_tablet_manager.cc:1397] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:08:26.531641 7693 ts_tablet_manager.cc:1397] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:08:26.531641 7692 ts_tablet_manager.cc:1397] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:08:26.534443 7691 raft_consensus.cc:348] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.534715 7691 raft_consensus.cc:374] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:26.534770 7691 raft_consensus.cc:729] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb13f63d46b4084838fe7c7c714a2db, State: Initialized, Role: FOLLOWER
I20260806 02:08:26.535218 7691 consensus_queue.cc:260] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.535403 7693 raft_consensus.cc:348] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.535403 7692 raft_consensus.cc:348] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.535653 7693 raft_consensus.cc:374] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:26.535653 7692 raft_consensus.cc:374] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:08:26.535712 7693 raft_consensus.cc:729] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fb3f80d684867906975e6fd7066e4, State: Initialized, Role: FOLLOWER
I20260806 02:08:26.535712 7692 raft_consensus.cc:729] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e9e93e740b4ba8bbbb62f78f6f7f32, State: Initialized, Role: FOLLOWER
I20260806 02:08:26.535991 7387 heartbeater.cc:503] Master 127.28.21.62:42495 was elected leader, sending a full tablet report...
I20260806 02:08:26.536121 7692 consensus_queue.cc:260] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.536121 7693 consensus_queue.cc:260] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.536260 7691 ts_tablet_manager.cc:1428] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:08:26.537003 7693 ts_tablet_manager.cc:1428] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:08:26.537003 7692 ts_tablet_manager.cc:1428] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:08:26.537338 7530 heartbeater.cc:503] Master 127.28.21.62:42495 was elected leader, sending a full tablet report...
I20260806 02:08:26.538201 7673 heartbeater.cc:503] Master 127.28.21.62:42495 was elected leader, sending a full tablet report...
I20260806 02:08:26.693866 7698 raft_consensus.cc:482] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:08:26.694116 7698 raft_consensus.cc:504] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.695468 7698 leader_election.cc:288] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fb3f80d684867906975e6fd7066e4 (127.28.21.3:42543), 67e9e93e740b4ba8bbbb62f78f6f7f32 (127.28.21.2:34467)
I20260806 02:08:26.699745 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.695832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:59739 (local address 127.28.21.3:42543)
0806 02:08:26.696003 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.696009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.696134 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:08:26.696295 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.696302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.696415 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.696502 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:26.696810 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.697376 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.698697 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.699135 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.699216 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.699266 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.699533 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.699610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:08:26.699771 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.695738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:42543 (local address 127.28.21.1:59739)
0806 02:08:26.696041 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.696055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.696169 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.696541 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.696545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.696566 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:26.696694 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.696700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.697501 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.697506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.698497 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.698508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.699256 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.699260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.699392 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.699422 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.699507 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.699589 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":155,"threads_started":1}
I20260806 02:08:26.699923 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.696353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:48845 (local address 127.28.21.2:34467)
0806 02:08:26.696482 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.696488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.696663 (+ 175us) server_negotiation.cc:407] Connection header received
0806 02:08:26.696784 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.696787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.696839 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.696918 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:08:26.697177 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.697659 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.698793 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.699294 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.699411 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.699483 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.699695 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.699768 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:08:26.700016 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.696228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:34467 (local address 127.28.21.1:48845)
0806 02:08:26.696582 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.696592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 02:08:26.696700 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:08:26.696930 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:08:26.696933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:08:26.696949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:08:26.697076 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.697080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.697765 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.697768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.698641 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:08:26.698654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.699420 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.699422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:08:26.699558 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.699587 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:08:26.699661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:08:26.699724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260806 02:08:26.700698 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef15dca467c4c23915a8757ab526ac1" candidate_uuid: "6bb13f63d46b4084838fe7c7c714a2db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" is_pre_election: true
I20260806 02:08:26.700701 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef15dca467c4c23915a8757ab526ac1" candidate_uuid: "6bb13f63d46b4084838fe7c7c714a2db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fb3f80d684867906975e6fd7066e4" is_pre_election: true
I20260806 02:08:26.701081 7481 raft_consensus.cc:2454] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb13f63d46b4084838fe7c7c714a2db in term 0.
I20260806 02:08:26.701081 7621 raft_consensus.cc:2454] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb13f63d46b4084838fe7c7c714a2db in term 0.
I20260806 02:08:26.701582 7269 leader_election.cc:302] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e93e740b4ba8bbbb62f78f6f7f32, 6bb13f63d46b4084838fe7c7c714a2db; no voters:
I20260806 02:08:26.701830 7698 raft_consensus.cc:2781] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:08:26.701926 7698 raft_consensus.cc:482] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:08:26.701987 7698 raft_consensus.cc:3037] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:26.703027 7698 raft_consensus.cc:504] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.703348 7698 leader_election.cc:288] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [CANDIDATE]: Term 1 election: Requested vote from peers 616fb3f80d684867906975e6fd7066e4 (127.28.21.3:42543), 67e9e93e740b4ba8bbbb62f78f6f7f32 (127.28.21.2:34467)
I20260806 02:08:26.703796 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef15dca467c4c23915a8757ab526ac1" candidate_uuid: "6bb13f63d46b4084838fe7c7c714a2db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32"
I20260806 02:08:26.703832 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef15dca467c4c23915a8757ab526ac1" candidate_uuid: "6bb13f63d46b4084838fe7c7c714a2db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fb3f80d684867906975e6fd7066e4"
I20260806 02:08:26.703924 7481 raft_consensus.cc:3037] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:26.703982 7621 raft_consensus.cc:3037] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:08:26.704989 7481 raft_consensus.cc:2454] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb13f63d46b4084838fe7c7c714a2db in term 1.
I20260806 02:08:26.705281 7269 leader_election.cc:302] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e9e93e740b4ba8bbbb62f78f6f7f32, 6bb13f63d46b4084838fe7c7c714a2db; no voters:
I20260806 02:08:26.705340 7621 raft_consensus.cc:2454] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb13f63d46b4084838fe7c7c714a2db in term 1.
I20260806 02:08:26.705443 7698 raft_consensus.cc:2781] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:08:26.705657 7698 raft_consensus.cc:686] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [term 1 LEADER]: Becoming Leader. State: Replica: 6bb13f63d46b4084838fe7c7c714a2db, State: Running, Role: LEADER
I20260806 02:08:26.705947 7698 consensus_queue.cc:237] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } }
I20260806 02:08:26.709061 7186 catalog_manager.cc:5495] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bb13f63d46b4084838fe7c7c714a2db (127.28.21.1). New cstate: current_term: 1 leader_uuid: "6bb13f63d46b4084838fe7c7c714a2db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb13f63d46b4084838fe7c7c714a2db" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 } health_report { overall_health: UNKNOWN } } }
W20260806 02:08:26.716188 7674 tablet.cc:2369] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:08:26.748297 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.745067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.28.21.1:36247)
0806 02:08:26.745190 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:08:26.745194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:08:26.745210 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:08:26.745286 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:08:26.745288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:08:26.745343 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:08:26.745427 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:08:26.745647 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.746142 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.747045 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:08:26.747249 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:08:26.747324 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:08:26.747583 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:08:26.747968 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:08:26.748068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:08:26.748142 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:08:26.748189 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:08:26.750506 7531 tablet.cc:2369] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:08:26.754202 7481 raft_consensus.cc:1261] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 1 FOLLOWER]: Refusing update from remote peer 6bb13f63d46b4084838fe7c7c714a2db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:26.754254 7621 raft_consensus.cc:1261] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 1 FOLLOWER]: Refusing update from remote peer 6bb13f63d46b4084838fe7c7c714a2db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:08:26.754899 7698 consensus_queue.cc:1035] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e9e93e740b4ba8bbbb62f78f6f7f32" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:26.755110 7704 consensus_queue.cc:1035] T eef15dca467c4c23915a8757ab526ac1 P 6bb13f63d46b4084838fe7c7c714a2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fb3f80d684867906975e6fd7066e4" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:08:26.761539 7707 mvcc.cc:205] Tried to move back new op lower bound from 7315382709258334208 to 7315382709070221312. Current Snapshot: MvccSnapshot[applied={T|T < 7315382709258334208}]
I20260806 02:08:26.764194 7709 mvcc.cc:205] Tried to move back new op lower bound from 7315382709258334208 to 7315382709070221312. Current Snapshot: MvccSnapshot[applied={T|T < 7315382709258334208}]
I20260806 02:08:26.770514 7186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:26.770659 7186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:08:26.772899 7186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 165103ac6c694048a0175d516c849a33: Sending DeleteTablet for 3 replicas of tablet eef15dca467c4c23915a8757ab526ac1
I20260806 02:08:26.773695 7319 tablet_service.cc:1514] Processing DeleteTablet for tablet eef15dca467c4c23915a8757ab526ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:26 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:58378
I20260806 02:08:26.773825 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet eef15dca467c4c23915a8757ab526ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:26 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:44794
I20260806 02:08:26.774189 7459 tablet_service.cc:1514] Processing DeleteTablet for tablet eef15dca467c4c23915a8757ab526ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:08:26 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:57184
I20260806 02:08:26.774349 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 7252
I20260806 02:08:26.775380 7719 tablet_replica.cc:330] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: stopping tablet replica
I20260806 02:08:26.775776 7719 raft_consensus.cc:2229] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:26.776154 7719 raft_consensus.cc:2258] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:26.777626 7719 ts_tablet_manager.cc:1905] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:08:26.779896 7719 ts_tablet_manager.cc:1918] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:08:26.780010 7719 log.cc:1196] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/eef15dca467c4c23915a8757ab526ac1
I20260806 02:08:26.780232 7718 tablet_replica.cc:330] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: stopping tablet replica
I20260806 02:08:26.780369 7719 ts_tablet_manager.cc:1939] T eef15dca467c4c23915a8757ab526ac1 P 67e9e93e740b4ba8bbbb62f78f6f7f32: Deleting consensus metadata
I20260806 02:08:26.780586 7718 raft_consensus.cc:2229] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:08:26.780905 7718 raft_consensus.cc:2258] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:08:26.781629 7169 catalog_manager.cc:4841] TS 67e9e93e740b4ba8bbbb62f78f6f7f32 (127.28.21.2:34467): tablet eef15dca467c4c23915a8757ab526ac1 (table test-table [id=fea344238fa14b01a95c60481e75a91d]) successfully deleted
I20260806 02:08:26.782323 7718 ts_tablet_manager.cc:1905] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:08:26.782888 7171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:36247 (error 108)
I20260806 02:08:26.782991 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 7392
I20260806 02:08:26.785738 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:08:26.785377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:36247 (local address 127.0.0.1:58392)
0806 02:08:26.785546 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:08:26.785629 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:36247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:08:26.785945 7718 ts_tablet_manager.cc:1918] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260806 02:08:26.785928 7171 catalog_manager.cc:4584] TS 6bb13f63d46b4084838fe7c7c714a2db (127.28.21.1:36247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eef15dca467c4c23915a8757ab526ac1: Network error: Client connection negotiation failed: client connection to 127.28.21.1:36247: connect: Connection refused (error 111)
I20260806 02:08:26.786062 7718 log.cc:1196] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/eef15dca467c4c23915a8757ab526ac1
I20260806 02:08:26.786369 7718 ts_tablet_manager.cc:1939] T eef15dca467c4c23915a8757ab526ac1 P 616fb3f80d684867906975e6fd7066e4: Deleting consensus metadata
I20260806 02:08:26.787439 7169 catalog_manager.cc:4841] TS 616fb3f80d684867906975e6fd7066e4 (127.28.21.3:42543): tablet eef15dca467c4c23915a8757ab526ac1 (table test-table [id=fea344238fa14b01a95c60481e75a91d]) successfully deleted
I20260806 02:08:26.789757 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 7535
I20260806 02:08:26.797457 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 7151
2026-08-06T02:08:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2528 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7723](info): set up 2 sockets
Aug 06 02:08:26 dist-test-slave-gp6n krb5kdc[7723](info): commencing operation
krb5kdc: starting...
W20260806 02:08:27.894161 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:08:27 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:08:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangeradmin_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangeradmin_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangerlookup_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangerlookup_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
I20260806 02:08:28.072764 28756 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRTVuz1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/postgres -l logfile start
2026-08-06 02:08:31.079 UTC [7755] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:08:31.079 UTC [7755] LOG: listening on IPv4 address "127.28.21.20", port 45443
2026-08-06 02:08:31.082 UTC [7755] LOG: listening on Unix socket "/tmp/.s.PGSQL.45443"
2026-08-06 02:08:31.103 UTC [7758] LOG: database system was shut down at 2026-08-06 02:08:29 UTC
2026-08-06 02:08:31.107 UTC [7755] LOG: database system is ready to accept connections
I20260806 02:08:32.100013 28756 mini_postgres.cc:96] Postgres bound to 45443
2026-08-06 02:08:32.108 UTC [7766] FATAL: database "slave" does not exist
127.28.21.20:45443 - accepting connections
I20260806 02:08:32.109133 28756 mini_ranger.cc:161] Starting Ranger...
I20260806 02:08:32.125540 28756 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:08:32.623309 28756 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:08:32.623458 28756 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin
2026-08-06 02:08:32,979 [I] DB FLAVOR :POSTGRES
2026-08-06 02:08:32,980 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:08:32,980 [I] Checking connection..
2026-08-06 02:08:32,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:08:33,257 [I] Checking connection passed.
2026-08-06 02:08:33,257 [I] --------- Verifying version history table ---------
2026-08-06 02:08:33,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:08:33,519 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:08:33,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:08:33,778 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:08:33,779 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:08:33,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:08:34.022 UTC [7833] WARNING: there is no transaction in progress
2026-08-06 02:08:34,029 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:08:34,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:08:34,304 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:08:34,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:08:34,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:08:34,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:08:34,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-06 02:08:35,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:08:35,297 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:08:35,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:08:35,545 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:08:35,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:08:35,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:08:36,048 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:08:36,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:08:37.045 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.062 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.079 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.097 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.115 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.136 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.198 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.206 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.222 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.233 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.243 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.258 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.268 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.275 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.282 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37.290 UTC [7946] WARNING: there is no transaction in progress
2026-08-06 02:08:37,974 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:08:37,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:08:38,233 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:08:38,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:08:38,513 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:08:38,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:08:38,778 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:08:38,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:08:39,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:08:39,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-06 02:08:39,490 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:08:39,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:08:39,748 [I] DB_PATCHES have already been applied
I20260806 02:08:39.754637 28756 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRTVuz1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:08:39.754715 28756 mini_ranger.cc:193] Using host: 127.28.21.20
I20260806 02:08:39.759111 28756 mini_ranger.cc:227] Ranger admin URL: http://127.28.21.20:54251
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:08:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:08:40.905987 28756 mini_ranger.cc:160] Time spent starting Ranger: real 8.797s user 0.003s sys 0.010s
Aug 06, 2026 2:09:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:09:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 2:09:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:09:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:09:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:09:03.789194 28756 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangerkms_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/rangerkms_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:09:03 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982143, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:09:03.883353 28756 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:09:03.894073 28756 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:09:04.824426 28756 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:09:04.824538 28756 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms
I20260806 02:09:04.824651 28756 mini_ranger_kms.cc:227] Using postgres at 127.28.21.20:45443
2026-08-06 02:09:05,064 [I] DB FLAVOR :POSTGRES
2026-08-06 02:09:05,064 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:09:05,064 [I] Checking connection
2026-08-06 02:09:05,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:09:05,336 [I] connection success
2026-08-06 02:09:05,337 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:09:05,337 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:09:05,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:09:05,581 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:09:05,581 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:09:05,581 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:09:05,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:45443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:09:05,840 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:09:06.353526 28756 mini_ranger_kms.cc:310] Created kms service
I20260806 02:09:06.527438 28756 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:09:06.578790 28756 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:09:06 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982143, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:07.041216 28756 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:09:07.041385 28756 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:09:07.041411 28756 mini_ranger_kms.cc:240] Using host: 127.28.21.20
I20260806 02:09:07.046079 28756 mini_ranger_kms.cc:276] Ranger KMS PID: 8141
I20260806 02:09:07.046243 28756 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.21.20:58487
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:09:09 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785982149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:09.470461 28756 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.587s user 0.007s sys 0.218s
I20260806 02:09:09.470613 28756 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:09:09.470736 28756 mini_ranger_kms.cc:401] 127.28.21.20:58487/kms/v1/keys
Aug 06 02:09:09 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
Aug 06 02:09:10 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
Aug 06 02:09:12 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:09:12.521054 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45311
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45311
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:58487/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:09:12.635212 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:12.635476 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:12.635515 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:12.639604 8195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:09:12.639669 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:12.639689 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:12.639706 8195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:09:12.639721 8195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:09:12.645215 8195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.21.20:58487/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45311
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:12.646333 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:12.647378 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:12.656651 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:12.656656 8195 server_base.cc:1034] running on GCE node
W20260806 02:09:12.656855 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:12.657001 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:12.657639 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:12.658850 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:12.660043 8195 hybrid_clock.cc:648] HybridClock initialized: now 1785982152660022 us; error 38 us; skew 500 ppm
Aug 06 02:09:12 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:12.667806 8195 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:09:12.669201 8195 webserver.cc:460] Webserver started at http://127.28.21.62:36605/ using document root <none> and password file <none>
I20260806 02:09:12.669862 8195 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:12.669947 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:12.670173 8195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:12 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:12.732949 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "386f8fc8402541e18eea1fba5a8fadb5"
format_stamp: "Formatted at 2026-08-06 02:09:12 on dist-test-slave-gp6n"
server_key: "8fad28a94a42489838c4b4b80a9c2f0e"
server_key_iv: "9bc38e1576b289dbe17f05f4c1624eb9"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:12.733623 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "386f8fc8402541e18eea1fba5a8fadb5"
format_stamp: "Formatted at 2026-08-06 02:09:12 on dist-test-slave-gp6n"
server_key: "8fad28a94a42489838c4b4b80a9c2f0e"
server_key_iv: "9bc38e1576b289dbe17f05f4c1624eb9"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:12.737788 8195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:09:12.755641 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:12.757148 8195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:09:12.757282 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "386f8fc8402541e18eea1fba5a8fadb5"
format_stamp: "Formatted at 2026-08-06 02:09:12 on dist-test-slave-gp6n"
server_key: "8fad28a94a42489838c4b4b80a9c2f0e"
server_key_iv: "9bc38e1576b289dbe17f05f4c1624eb9"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:12.757422 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:09:12.772503 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:09:12.775740 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:09:12.775950 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:09:12.784618 8195 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:45311
I20260806 02:09:12.784669 8267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:45311 every 8 connection(s)
I20260806 02:09:12.785626 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:09:12.788321 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 8195
I20260806 02:09:12.788462 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:09:12.788727 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:09:12.788803 28756 external_mini_cluster.cc:1428] Setting key a5870283606862b212ee9e9220b60524
I20260806 02:09:12.795964 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: Bootstrap starting.
I20260806 02:09:12.798699 8268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: Neither blocks nor log segments found. Creating new log.
I20260806 02:09:12.799492 8268 log.cc:824] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:09:12 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:09:12.801813 8268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: No bootstrap required, opened a new log
I20260806 02:09:12.805402 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:12.790273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.28.21.62:45311)
0806 02:09:12.790765 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:12.790784 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:12.790827 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:09:12.791676 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:12.791700 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:12.792131 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:12.792491 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:12.793404 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:12.795008 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:12.796216 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:12.796894 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:12.797065 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:12.800206 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:12.800213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:12.800231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:12.800265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:12.802625 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:12.803088 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:12.803093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:12.803097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:12.803201 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:12.803495 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:12.803499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:12.803501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:12.803888 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:12.804101 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:12.804385 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:09:12.804684 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":170,"threads_started":1}
I20260806 02:09:12.805757 8268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } }
I20260806 02:09:12.806041 8268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:09:12.806123 8268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f8fc8402541e18eea1fba5a8fadb5, State: Initialized, Role: FOLLOWER
I20260806 02:09:12.806630 8268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } }
I20260806 02:09:12.806787 8268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:09:12.806872 8268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:09:12.806974 8268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:09:12.808068 8268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } }
I20260806 02:09:12.808408 8268 leader_election.cc:302] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 386f8fc8402541e18eea1fba5a8fadb5; no voters:
I20260806 02:09:12.808717 8268 leader_election.cc:288] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:09:12.808876 8273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:09:12.809171 8273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [term 1 LEADER]: Becoming Leader. State: Replica: 386f8fc8402541e18eea1fba5a8fadb5, State: Running, Role: LEADER
I20260806 02:09:12.809513 8273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } }
I20260806 02:09:12.810097 8268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:09:12.810701 8273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 386f8fc8402541e18eea1fba5a8fadb5. Latest consensus state: current_term: 1 leader_uuid: "386f8fc8402541e18eea1fba5a8fadb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } } }
I20260806 02:09:12.810932 8273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:09:12.811234 8274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "386f8fc8402541e18eea1fba5a8fadb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "386f8fc8402541e18eea1fba5a8fadb5" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45311 } } }
I20260806 02:09:12.811390 8274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:09:12.811395 8283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:09:12.815016 8283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:09:12.815626 8268 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:09:12.821020 8283 catalog_manager.cc:1347] Generated new cluster ID: e1ad7659841844f7a377d2b785f07031
I20260806 02:09:12.821190 8283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:09:12.865407 8283 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:09:12.866863 8283 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:09:12.872597 8283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: Generated new TSK 0
I20260806 02:09:12.873629 8283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:09:13.689256 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:09:13.857038 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:13.857519 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:13.857653 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:13.863869 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:13.864013 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:13.864153 8318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:09:13.870741 8318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.21.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:13.872394 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:13.873740 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:13.888463 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:13.888489 8328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:13.888937 8325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:13.889979 8318 server_base.cc:1034] running on GCE node
I20260806 02:09:13.890491 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:13.891278 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:13.892464 8318 hybrid_clock.cc:648] HybridClock initialized: now 1785982153892429 us; error 56 us; skew 500 ppm
Aug 06 02:09:13 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:13.895753 8318 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:09:13.896812 8318 webserver.cc:460] Webserver started at http://127.28.21.1:45051/ using document root <none> and password file <none>
I20260806 02:09:13.897400 8318 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:13.897496 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:13.897746 8318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:13 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:13.915159 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "37a2c907ae104a0892d4945cd289aa12"
format_stamp: "Formatted at 2026-08-06 02:09:13 on dist-test-slave-gp6n"
server_key: "6d5b06cbd56702a4f94705a05b99d6df"
server_key_iv: "b2f6f3d4853af83d69eea06afb53a141"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:13.918713 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "37a2c907ae104a0892d4945cd289aa12"
format_stamp: "Formatted at 2026-08-06 02:09:13 on dist-test-slave-gp6n"
server_key: "6d5b06cbd56702a4f94705a05b99d6df"
server_key_iv: "b2f6f3d4853af83d69eea06afb53a141"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:13.924559 8318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:09:13.954123 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:13.955850 8318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:09:13.956012 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "37a2c907ae104a0892d4945cd289aa12"
format_stamp: "Formatted at 2026-08-06 02:09:13 on dist-test-slave-gp6n"
server_key: "6d5b06cbd56702a4f94705a05b99d6df"
server_key_iv: "b2f6f3d4853af83d69eea06afb53a141"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:13.956156 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:09:13.977109 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:09:13.988889 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:09:13.989108 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:09:13.989886 8318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:09:13.991010 8318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:09:13.991077 8318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:13.991140 8318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:09:13.991209 8318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:09:14.002012 8318 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:42951
I20260806 02:09:14.003218 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:09:14.005259 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 8318
I20260806 02:09:14.005427 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:09:14.005759 28756 external_mini_cluster.cc:1428] Setting key 47712ce1ff4d288ed36d2f8a71b3fcf5
I20260806 02:09:14.010335 8462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:42951 every 8 connection(s)
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:09:14.022055 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.007431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:42963 (local address 127.28.21.62:45311)
0806 02:09:14.009391 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.009395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.009409 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:09:14.009448 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.009451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.009495 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.009581 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:14.011184 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.012051 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.013791 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.014152 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.014239 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.017479 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:14.017483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:14.017486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:14.017513 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:14.019459 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.020099 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.020102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.020104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.020202 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.020678 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.020681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.020683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.020912 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:14.021010 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.021729 (+ 719us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.021867 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:09:14.023761 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.006200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45311 (local address 127.28.21.1:42963)
0806 02:09:14.007858 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.007916 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.008943 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.009895 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.009905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.010271 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:09:14.011011 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.011032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.012201 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.012204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.013644 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.013656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.014500 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.014503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.014706 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.015383 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:09:14.015404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:09:14.017265 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:09:14.019608 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.019614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.019631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.019975 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.020324 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.020327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.020329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.020531 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.021080 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:09:14.021086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:09:14.021433 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.022304 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.022711 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"thread_start_us":183,"threads_started":1}
I20260806 02:09:14.025384 8463 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45311
I20260806 02:09:14.025640 8463 heartbeater.cc:464] Registering TS with master...
I20260806 02:09:14.026142 8463 heartbeater.cc:511] Master 127.28.21.62:45311 requested a full tablet report, sending...
I20260806 02:09:14.027971 8230 ts_manager.cc:194] Registered new tserver with Master: 37a2c907ae104a0892d4945cd289aa12 (127.28.21.1:42951)
I20260806 02:09:14.029652 8230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:42963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:09:14.042565 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:09:14.174829 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:14.175199 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:14.175323 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:14.179705 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:14.179790 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:14.179895 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:09:14.185415 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.21.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:14.186652 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:14.187774 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:14.198501 8478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:14.198499 8476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:14.198767 8475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:14.199553 8469 server_base.cc:1034] running on GCE node
I20260806 02:09:14.199970 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:14.200685 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:14.201875 8469 hybrid_clock.cc:648] HybridClock initialized: now 1785982154201844 us; error 56 us; skew 500 ppm
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:14.209393 8469 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:09:14.210661 8469 webserver.cc:460] Webserver started at http://127.28.21.2:42063/ using document root <none> and password file <none>
I20260806 02:09:14.211551 8469 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:14.211649 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:14.212002 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:14.229151 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "41768d15440b4260a439dad0ce85f58c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "4815ec2cbd3ae09b3b403fcdb6f34872"
server_key_iv: "637c52f303522c8cacd5b307ec746c28"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.229889 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "41768d15440b4260a439dad0ce85f58c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "4815ec2cbd3ae09b3b403fcdb6f34872"
server_key_iv: "637c52f303522c8cacd5b307ec746c28"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.235442 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:09:14.258991 8485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.260908 8469 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 02:09:14.261050 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "41768d15440b4260a439dad0ce85f58c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "4815ec2cbd3ae09b3b403fcdb6f34872"
server_key_iv: "637c52f303522c8cacd5b307ec746c28"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.261190 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:09:14.282946 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:09:14.286972 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:09:14.287258 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:09:14.288081 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:09:14.289266 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:09:14.289435 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.289551 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:09:14.289633 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.302129 8469 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:41409
I20260806 02:09:14.303675 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:09:14.307824 8606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:41409 every 8 connection(s)
I20260806 02:09:14.312263 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 8469
I20260806 02:09:14.312404 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:09:14.312740 28756 external_mini_cluster.cc:1428] Setting key 623fc6069710cab1116a15e79cd96258
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:09:14.339671 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.310047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:36279 (local address 127.28.21.62:45311)
0806 02:09:14.310217 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.310222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.314910 (+ 4688us) server_negotiation.cc:407] Connection header received
0806 02:09:14.316471 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.316478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.316542 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.316605 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:14.318374 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.319286 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.322581 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.322966 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.323070 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.335132 (+ 12062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:14.335137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:14.335140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:14.335176 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:14.337057 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.337729 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.337733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.337735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.337825 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.338255 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.338258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.338260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.338521 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:14.338608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.339348 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.339492 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:09:14.341624 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.309176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45311 (local address 127.28.21.2:36279)
0806 02:09:14.314740 (+ 5564us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.314806 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.316210 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.317082 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.317094 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.317473 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:09:14.318199 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.318223 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.319462 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.319468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.321047 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.321061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.330345 (+ 9284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.330351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.330681 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.331508 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:09:14.331536 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:09:14.334881 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:09:14.337213 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.337220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.337239 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.337600 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.337933 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.337936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.337938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.338136 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.338676 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:09:14.338683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:09:14.339051 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.340167 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.340533 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5322,"thread_start_us":156,"threads_started":1}
I20260806 02:09:14.348086 8607 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45311
I20260806 02:09:14.348417 8607 heartbeater.cc:464] Registering TS with master...
I20260806 02:09:14.349128 8607 heartbeater.cc:511] Master 127.28.21.62:45311 requested a full tablet report, sending...
I20260806 02:09:14.350505 8230 ts_manager.cc:194] Registered new tserver with Master: 41768d15440b4260a439dad0ce85f58c (127.28.21.2:41409)
I20260806 02:09:14.351289 8230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:36279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:09:14.358811 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:09:14.528788 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:14.529273 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:14.529428 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:14.536029 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:14.536171 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:14.536303 8613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:09:14.544137 8613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.21.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:14.545760 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:14.547233 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:14.560563 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:14.561103 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:14.561389 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:14.563581 8613 server_base.cc:1034] running on GCE node
I20260806 02:09:14.564033 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:14.564769 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:14.580863 8613 hybrid_clock.cc:648] HybridClock initialized: now 1785982154580768 us; error 116 us; skew 500 ppm
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:14.587706 8613 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:09:14.588840 8613 webserver.cc:460] Webserver started at http://127.28.21.3:37241/ using document root <none> and password file <none>
I20260806 02:09:14.589583 8613 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:14.589672 8613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:14.589890 8613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:14.618788 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "36d54fd811f94119818f82847ab19e6c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "f76f4bd82c4d6a97038c50e0b60c254f"
server_key_iv: "740ff4dedb2a40adae650de7b933e9f1"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.622033 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "36d54fd811f94119818f82847ab19e6c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "f76f4bd82c4d6a97038c50e0b60c254f"
server_key_iv: "740ff4dedb2a40adae650de7b933e9f1"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.634071 8613 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.000s sys 0.004s
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:14.654686 8629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.656375 8613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:09:14.656524 8613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "36d54fd811f94119818f82847ab19e6c"
format_stamp: "Formatted at 2026-08-06 02:09:14 on dist-test-slave-gp6n"
server_key: "f76f4bd82c4d6a97038c50e0b60c254f"
server_key_iv: "740ff4dedb2a40adae650de7b933e9f1"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:14.656672 8613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:09:14.701794 8613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:09:14.705928 8613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:09:14.706209 8613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:09:14.706997 8613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:09:14.708163 8613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:09:14.708277 8613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.708388 8613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:09:14.708461 8613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:14.720376 8613 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:33639
I20260806 02:09:14.721906 8613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:09:14.727689 8747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:33639 every 8 connection(s)
I20260806 02:09:14.730157 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 8613
I20260806 02:09:14.730284 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:09:14.730626 28756 external_mini_cluster.cc:1428] Setting key dd4561f2066740bd29a67aca9c260f65
Aug 06 02:09:14 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:09:14.747619 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.729631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:47923 (local address 127.28.21.62:45311)
0806 02:09:14.729764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.729769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.733850 (+ 4081us) server_negotiation.cc:407] Connection header received
0806 02:09:14.735123 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.735126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.735183 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.735239 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:14.736982 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.737818 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.739568 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.739881 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.739960 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.743179 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:14.743183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:14.743186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:14.743213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:14.745020 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.745684 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.745688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.745690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.745780 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.746202 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.746205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.746207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.746481 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:14.746571 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.747297 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.747439 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:09:14.749243 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.730142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45311 (local address 127.28.21.3:47923)
0806 02:09:14.733695 (+ 3553us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.733756 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.734870 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.735706 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.735716 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.736087 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:09:14.736800 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.736823 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.737996 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.737999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.739426 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.739438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.740175 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.740179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.740408 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.741185 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:09:14.741207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:09:14.742972 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:09:14.745168 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.745174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.745191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.745559 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.745897 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:09:14.745900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:09:14.745903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:09:14.746082 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:09:14.746642 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:09:14.746649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:09:14.746997 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.747805 (+ 808us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.748150 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3319,"thread_start_us":177,"threads_started":1}
I20260806 02:09:14.750840 8748 heartbeater.cc:347] Connected to a master server at 127.28.21.62:45311
I20260806 02:09:14.751075 8748 heartbeater.cc:464] Registering TS with master...
I20260806 02:09:14.751602 8748 heartbeater.cc:511] Master 127.28.21.62:45311 requested a full tablet report, sending...
I20260806 02:09:14.752677 8230 ts_manager.cc:194] Registered new tserver with Master: 36d54fd811f94119818f82847ab19e6c (127.28.21.3:33639)
I20260806 02:09:14.753367 8230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:47923
I20260806 02:09:14.756841 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:09:14.796175 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.764153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.28.21.62:45311)
0806 02:09:14.764268 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.764272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.774747 (+ 10475us) server_negotiation.cc:407] Connection header received
0806 02:09:14.774968 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.774972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.775023 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.775075 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:14.776126 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.776974 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.778205 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.778585 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.778662 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.780877 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:14.780881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:14.780884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:14.780910 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:14.782708 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.794538 (+ 11830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.794543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.794545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.794678 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:14.795072 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:14.795076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:14.795078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:14.795329 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:14.795452 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.795851 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.795994 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:09:14.800405 8230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:09:14.805078 8230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:09:14.831210 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.823214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42951 (local address 127.0.0.1:46244)
0806 02:09:14.825310 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.825323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.825411 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.826179 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.826185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.826208 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:09:14.826339 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.826346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.827810 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.827814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.829216 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.829228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.830636 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.830640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.830754 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.830795 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.830922 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.831050 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2043,"thread_start_us":78,"threads_started":1}
I20260806 02:09:14.831851 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.823649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.28.21.1:42951)
0806 02:09:14.824696 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.824706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.825507 (+ 801us) server_negotiation.cc:407] Connection header received
0806 02:09:14.825591 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.825597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.825849 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.826001 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:09:14.826518 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.827676 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.829385 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.830270 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.830374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.830571 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.831592 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.831698 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":105,"threads_started":1}
I20260806 02:09:14.835927 8392 tablet_service.cc:1467] Processing CreateTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 (DEFAULT_TABLE table=test-table [id=27cde9a2dd8f4d15ab696e9dab078f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:09:14.837297 8392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeda9f29ef32454ca3c8b3d55db34922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:09:14.852618 8768 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Bootstrap starting.
I20260806 02:09:14.855531 8768 tablet_bootstrap.cc:654] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Neither blocks nor log segments found. Creating new log.
I20260806 02:09:14.856637 8768 log.cc:824] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Log is configured to *not* fsync() on all Append() calls
I20260806 02:09:14.859110 8768 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: No bootstrap required, opened a new log
I20260806 02:09:14.859328 8768 ts_tablet_manager.cc:1397] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:09:14.864526 8768 raft_consensus.cc:348] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.865228 8768 raft_consensus.cc:374] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:09:14.865345 8768 raft_consensus.cc:729] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a2c907ae104a0892d4945cd289aa12, State: Initialized, Role: FOLLOWER
I20260806 02:09:14.865972 8768 consensus_queue.cc:260] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.867329 8768 ts_tablet_manager.cc:1428] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:09:14.867792 8463 heartbeater.cc:503] Master 127.28.21.62:45311 was elected leader, sending a full tablet report...
I20260806 02:09:14.875892 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.822930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:41409 (local address 127.0.0.1:50038)
0806 02:09:14.824891 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.824949 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.825092 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.845274 (+ 20182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.845278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.845299 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:09:14.845442 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.845450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.870535 (+ 25085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.870540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.871739 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.871754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.875439 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.875443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.875550 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.875580 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.875630 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.875714 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"spinlock_wait_cycles":6656,"thread_start_us":83,"threads_started":1}
I20260806 02:09:14.876909 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.823105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.28.21.2:41409)
0806 02:09:14.843993 (+ 20888us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.844013 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.844047 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:09:14.844136 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.844141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.844381 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.845632 (+ 1251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:09:14.845792 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.847005 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.871954 (+ 24949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.872859 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.876401 (+ 3542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.876595 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.876656 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.876764 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20758,"thread_start_us":93,"threads_started":1}
I20260806 02:09:14.877622 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.820590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:33639 (local address 127.0.0.1:53392)
0806 02:09:14.824891 (+ 4301us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.824950 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.825092 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.847188 (+ 22096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.847192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.847214 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:09:14.847334 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.847341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.873028 (+ 25687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.873032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.874134 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.874148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.877252 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.877256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.877349 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.877374 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:09:14.877426 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:09:14.877495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4114,"spinlock_wait_cycles":4736,"thread_start_us":103,"threads_started":1}
I20260806 02:09:14.878549 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.821563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.28.21.3:33639)
0806 02:09:14.844744 (+ 23181us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.844758 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.844784 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:09:14.844881 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.844888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.845085 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.851197 (+ 6112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:09:14.851380 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.862145 (+ 10765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.874371 (+ 12226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.875272 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.878052 (+ 2780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:14.878216 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:14.878270 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:09:14.878366 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23048,"thread_start_us":132,"threads_started":1}
I20260806 02:09:14.882043 8522 tablet_service.cc:1467] Processing CreateTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 (DEFAULT_TABLE table=test-table [id=27cde9a2dd8f4d15ab696e9dab078f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:09:14.882484 8666 tablet_service.cc:1467] Processing CreateTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 (DEFAULT_TABLE table=test-table [id=27cde9a2dd8f4d15ab696e9dab078f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:09:14.883230 8522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeda9f29ef32454ca3c8b3d55db34922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:09:14.883463 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeda9f29ef32454ca3c8b3d55db34922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:09:14.894789 8771 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Bootstrap starting.
I20260806 02:09:14.897387 8771 tablet_bootstrap.cc:654] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Neither blocks nor log segments found. Creating new log.
I20260806 02:09:14.901639 8771 log.cc:824] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:09:14.902513 8772 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Bootstrap starting.
I20260806 02:09:14.905474 8772 tablet_bootstrap.cc:654] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Neither blocks nor log segments found. Creating new log.
I20260806 02:09:14.906375 8772 log.cc:824] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:09:14.908694 8771 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: No bootstrap required, opened a new log
I20260806 02:09:14.908900 8771 ts_tablet_manager.cc:1397] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 02:09:14.914076 8771 raft_consensus.cc:348] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.914439 8771 raft_consensus.cc:374] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:09:14.914536 8771 raft_consensus.cc:729] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41768d15440b4260a439dad0ce85f58c, State: Initialized, Role: FOLLOWER
I20260806 02:09:14.915064 8771 consensus_queue.cc:260] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.916190 8771 ts_tablet_manager.cc:1428] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:09:14.916589 8607 heartbeater.cc:503] Master 127.28.21.62:45311 was elected leader, sending a full tablet report...
I20260806 02:09:14.921182 8772 tablet_bootstrap.cc:492] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: No bootstrap required, opened a new log
I20260806 02:09:14.921453 8772 ts_tablet_manager.cc:1397] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20260806 02:09:14.925770 8772 raft_consensus.cc:348] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.926046 8772 raft_consensus.cc:374] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:09:14.926088 8772 raft_consensus.cc:729] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d54fd811f94119818f82847ab19e6c, State: Initialized, Role: FOLLOWER
I20260806 02:09:14.926788 8772 consensus_queue.cc:260] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.927932 8772 ts_tablet_manager.cc:1428] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 02:09:14.928315 8748 heartbeater.cc:503] Master 127.28.21.62:45311 was elected leader, sending a full tablet report...
W20260806 02:09:14.979719 8749 tablet.cc:2369] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:09:14.992594 8776 raft_consensus.cc:482] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:09:14.992851 8776 raft_consensus.cc:504] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:14.994197 8776 leader_election.cc:288] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d54fd811f94119818f82847ab19e6c (127.28.21.3:33639), 37a2c907ae104a0892d4945cd289aa12 (127.28.21.1:42951)
I20260806 02:09:15.001011 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.995797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:60829 (local address 127.28.21.3:33639)
0806 02:09:14.995921 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.995926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.996095 (+ 169us) server_negotiation.cc:407] Connection header received
0806 02:09:14.996287 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.996291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.996356 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.996404 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:09:14.996814 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.997568 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.999553 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.000109 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.000187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:15.000248 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:15.000806 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:09:15.000884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:09:15.001637 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.994620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:33639 (local address 127.28.21.2:60829)
0806 02:09:14.996003 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.996018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.996175 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:14.996546 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:14.996549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:14.996580 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:09:14.996691 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.996697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:14.997735 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:14.997738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:14.999052 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:14.999064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.000591 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.000593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:15.000680 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:15.000702 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:09:15.001397 (+ 695us) client_negotiation.cc:240] Negotiation successful
0806 02:09:15.001475 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":120,"threads_started":1}
I20260806 02:09:15.002569 8678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeda9f29ef32454ca3c8b3d55db34922" candidate_uuid: "41768d15440b4260a439dad0ce85f58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d54fd811f94119818f82847ab19e6c" is_pre_election: true
I20260806 02:09:15.002928 8678 raft_consensus.cc:2454] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41768d15440b4260a439dad0ce85f58c in term 0.
I20260806 02:09:15.003575 8488 leader_election.cc:302] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d54fd811f94119818f82847ab19e6c, 41768d15440b4260a439dad0ce85f58c; no voters:
I20260806 02:09:15.003906 8776 raft_consensus.cc:2781] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:09:15.003988 8776 raft_consensus.cc:482] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:09:15.004027 8776 raft_consensus.cc:3037] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:09:15.005445 8776 raft_consensus.cc:504] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:15.005837 8776 leader_election.cc:288] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [CANDIDATE]: Term 1 election: Requested vote from peers 36d54fd811f94119818f82847ab19e6c (127.28.21.3:33639), 37a2c907ae104a0892d4945cd289aa12 (127.28.21.1:42951)
I20260806 02:09:15.006610 8678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeda9f29ef32454ca3c8b3d55db34922" candidate_uuid: "41768d15440b4260a439dad0ce85f58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d54fd811f94119818f82847ab19e6c"
I20260806 02:09:15.006744 8678 raft_consensus.cc:3037] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:09:15.008196 8678 raft_consensus.cc:2454] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41768d15440b4260a439dad0ce85f58c in term 1.
W20260806 02:09:15.011049 8464 tablet.cc:2369] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:09:15.013150 8488 leader_election.cc:302] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d54fd811f94119818f82847ab19e6c, 41768d15440b4260a439dad0ce85f58c; no voters:
I20260806 02:09:15.013424 8776 raft_consensus.cc:2781] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:09:15.013705 8776 raft_consensus.cc:686] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 1 LEADER]: Becoming Leader. State: Replica: 41768d15440b4260a439dad0ce85f58c, State: Running, Role: LEADER
I20260806 02:09:15.014065 8776 consensus_queue.cc:237] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } }
I20260806 02:09:15.024142 8230 catalog_manager.cc:5495] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c reported cstate change: term changed from 0 to 1, leader changed from <none> to 41768d15440b4260a439dad0ce85f58c (127.28.21.2). New cstate: current_term: 1 leader_uuid: "41768d15440b4260a439dad0ce85f58c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41768d15440b4260a439dad0ce85f58c" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 41409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 } health_report { overall_health: UNKNOWN } } }
I20260806 02:09:15.031095 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.995503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:57051 (local address 127.28.21.1:42951)
0806 02:09:14.995691 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:14.995696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:14.999419 (+ 3723us) server_negotiation.cc:407] Connection header received
0806 02:09:14.999530 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:14.999534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:14.999588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:14.999684 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:09:15.000563 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.001291 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.029650 (+ 28359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.030299 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.030436 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:15.030507 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:15.030878 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:09:15.030965 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:09:15.031752 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:14.995032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42951 (local address 127.28.21.2:57051)
0806 02:09:14.999329 (+ 4297us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:14.999342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:09:14.999433 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:09:15.000312 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:09:15.000315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:09:15.000334 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:09:15.000452 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:15.000458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.001446 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.001450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:15.029456 (+ 28006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:09:15.029473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.030614 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.030618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:09:15.030727 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:15.030751 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:09:15.031504 (+ 753us) client_negotiation.cc:240] Negotiation successful
0806 02:09:15.031596 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4217,"thread_start_us":95,"threads_started":1}
I20260806 02:09:15.032570 8413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeda9f29ef32454ca3c8b3d55db34922" candidate_uuid: "41768d15440b4260a439dad0ce85f58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a2c907ae104a0892d4945cd289aa12" is_pre_election: true
I20260806 02:09:15.032920 8413 raft_consensus.cc:2454] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41768d15440b4260a439dad0ce85f58c in term 0.
I20260806 02:09:15.033290 8412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeda9f29ef32454ca3c8b3d55db34922" candidate_uuid: "41768d15440b4260a439dad0ce85f58c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a2c907ae104a0892d4945cd289aa12"
I20260806 02:09:15.033416 8412 raft_consensus.cc:3037] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:09:15.034865 8412 raft_consensus.cc:2454] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41768d15440b4260a439dad0ce85f58c in term 1.
W20260806 02:09:15.061229 8608 tablet.cc:2369] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:09:15 dist-test-slave-gp6n krb5kdc[7723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:15.073233 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:15.068285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.28.21.2:41409)
0806 02:09:15.068432 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:15.068437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:15.068455 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:09:15.068544 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:15.068547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:15.068619 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:15.068740 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:09:15.069156 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.069903 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.071159 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:15.071507 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:15.071598 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:15.071994 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:09:15.072539 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:09:15.072653 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:15.073031 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:09:15.073092 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:09:15.086179 8412 raft_consensus.cc:1261] T eeda9f29ef32454ca3c8b3d55db34922 P 37a2c907ae104a0892d4945cd289aa12 [term 1 FOLLOWER]: Refusing update from remote peer 41768d15440b4260a439dad0ce85f58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:09:15.086179 8678 raft_consensus.cc:1261] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 1 FOLLOWER]: Refusing update from remote peer 41768d15440b4260a439dad0ce85f58c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:09:15.087004 8776 consensus_queue.cc:1035] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d54fd811f94119818f82847ab19e6c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:09:15.087435 8776 consensus_queue.cc:1035] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a2c907ae104a0892d4945cd289aa12" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:09:15.096508 8784 mvcc.cc:205] Tried to move back new op lower bound from 7315382907223609344 to 7315382906945486848. Current Snapshot: MvccSnapshot[applied={T|T < 7315382907223609344}]
I20260806 02:09:15.102375 8785 mvcc.cc:205] Tried to move back new op lower bound from 7315382907223609344 to 7315382906945486848. Current Snapshot: MvccSnapshot[applied={T|T < 7315382907223609344}]
I20260806 02:09:15.112694 8227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:09:15.112886 8227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:09:15.113564 8787 mvcc.cc:205] Tried to move back new op lower bound from 7315382907223609344 to 7315382906945486848. Current Snapshot: MvccSnapshot[applied={T|T < 7315382907223609344}]
I20260806 02:09:15.115959 8227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 386f8fc8402541e18eea1fba5a8fadb5: Sending DeleteTablet for 3 replicas of tablet eeda9f29ef32454ca3c8b3d55db34922
I20260806 02:09:15.117162 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:09:15 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260806 02:09:15.117650 8522 tablet_service.cc:1514] Processing DeleteTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:09:15 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:50038
I20260806 02:09:15.118634 8392 tablet_service.cc:1514] Processing DeleteTablet for tablet eeda9f29ef32454ca3c8b3d55db34922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:09:15 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260806 02:09:15.119926 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 8318
I20260806 02:09:15.124691 8793 tablet_replica.cc:330] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: stopping tablet replica
I20260806 02:09:15.124953 8794 tablet_replica.cc:330] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: stopping tablet replica
I20260806 02:09:15.125375 8794 raft_consensus.cc:2229] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:09:15.125746 8793 raft_consensus.cc:2229] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:09:15.126165 8793 raft_consensus.cc:2258] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:09:15.126610 8794 raft_consensus.cc:2258] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:09:15.128545 8794 ts_tablet_manager.cc:1905] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:09:15.129916 8793 ts_tablet_manager.cc:1905] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:09:15.132824 8793 ts_tablet_manager.cc:1918] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:09:15.135685 8794 ts_tablet_manager.cc:1918] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:09:15.135867 8793 log.cc:1196] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/eeda9f29ef32454ca3c8b3d55db34922
I20260806 02:09:15.136021 8794 log.cc:1196] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/eeda9f29ef32454ca3c8b3d55db34922
I20260806 02:09:15.136535 8793 ts_tablet_manager.cc:1939] T eeda9f29ef32454ca3c8b3d55db34922 P 36d54fd811f94119818f82847ab19e6c: Deleting consensus metadata
I20260806 02:09:15.136915 8794 ts_tablet_manager.cc:1939] T eeda9f29ef32454ca3c8b3d55db34922 P 41768d15440b4260a439dad0ce85f58c: Deleting consensus metadata
I20260806 02:09:15.138446 8214 catalog_manager.cc:4841] TS 41768d15440b4260a439dad0ce85f58c (127.28.21.2:41409): tablet eeda9f29ef32454ca3c8b3d55db34922 (table test-table [id=27cde9a2dd8f4d15ab696e9dab078f6d]) successfully deleted
I20260806 02:09:15.139981 8214 catalog_manager.cc:4841] TS 36d54fd811f94119818f82847ab19e6c (127.28.21.3:33639): tablet eeda9f29ef32454ca3c8b3d55db34922 (table test-table [id=27cde9a2dd8f4d15ab696e9dab078f6d]) successfully deleted
W20260806 02:09:15.142699 8214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:42951 (error 108)
I20260806 02:09:15.144426 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:15.144141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42951 (local address 127.0.0.1:46248)
0806 02:09:15.144262 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 02:09:15.144338 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 02:09:15.144413 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 8469
W20260806 02:09:15.151767 8214 catalog_manager.cc:4584] TS 37a2c907ae104a0892d4945cd289aa12 (127.28.21.1:42951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eeda9f29ef32454ca3c8b3d55db34922: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42951: connect: Connection refused (error 111)
I20260806 02:09:15.159283 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 8613
I20260806 02:09:15.168769 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 8195
I20260806 02:09:15.178486 28756 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:09:15.523494 28756 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:09:15.523614 28756 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:09:15.868348 28756 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:09:15.868 UTC [7755] LOG: received smart shutdown request
2026-08-06 02:09:15.872 UTC [7755] LOG: background worker "logical replication launcher" (PID 7764) exited with exit code 1
2026-08-06 02:09:15.873 UTC [7759] LOG: shutting down
2026-08-06 02:09:16.050 UTC [7755] LOG: database system is shut down
2026-08-06T02:09:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49370 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:09:16 dist-test-slave-gp6n krb5kdc[8807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:09:16 dist-test-slave-gp6n krb5kdc[8807](info): set up 2 sockets
Aug 06 02:09:16 dist-test-slave-gp6n krb5kdc[8807](info): commencing operation
krb5kdc: starting...
W20260806 02:09:17.328848 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:09:17 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:09:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangeradmin_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangeradmin_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangerlookup_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangerlookup_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
I20260806 02:09:17.505533 28756 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRTVuz1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/postgres -l logfile start
2026-08-06 02:09:20.060 UTC [8837] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:09:20.060 UTC [8837] LOG: listening on IPv4 address "127.28.21.20", port 59495
2026-08-06 02:09:20.065 UTC [8837] LOG: listening on Unix socket "/tmp/.s.PGSQL.59495"
2026-08-06 02:09:20.088 UTC [8840] LOG: database system was shut down at 2026-08-06 02:09:18 UTC
2026-08-06 02:09:20.093 UTC [8837] LOG: database system is ready to accept connections
I20260806 02:09:21.073062 28756 mini_postgres.cc:96] Postgres bound to 59495
2026-08-06 02:09:21.080 UTC [8848] FATAL: database "slave" does not exist
127.28.21.20:59495 - accepting connections
I20260806 02:09:21.080701 28756 mini_ranger.cc:161] Starting Ranger...
I20260806 02:09:21.096246 28756 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:09:21.584242 28756 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:09:21.584345 28756 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin
2026-08-06 02:09:21,861 [I] DB FLAVOR :POSTGRES
2026-08-06 02:09:21,862 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:09:21,862 [I] Checking connection..
2026-08-06 02:09:21,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:09:22,129 [I] Checking connection passed.
2026-08-06 02:09:22,129 [I] --------- Verifying version history table ---------
2026-08-06 02:09:22,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:09:22,377 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:09:22,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:09:22,605 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:09:22,605 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:09:22,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:09:22.829 UTC [8917] WARNING: there is no transaction in progress
2026-08-06 02:09:22,836 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:09:22,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:09:23,083 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:09:23,084 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:09:23,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:09:23,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:09:23,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-06 02:09:23,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:09:24,027 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:09:24,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:09:24,266 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:09:24,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:09:24,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:09:24,753 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:09:24,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:09:25.722 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.739 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.755 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.771 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.790 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.808 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.872 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.879 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.894 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.903 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.913 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.928 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.938 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.945 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.952 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:25.959 UTC [9029] WARNING: there is no transaction in progress
2026-08-06 02:09:26,639 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:09:26,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:09:26,891 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:09:26,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:09:27,145 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:09:27,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:09:27,403 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:09:27,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:09:27,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:09:27,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-06 02:09:28,116 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:09:28,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:09:28,374 [I] DB_PATCHES have already been applied
I20260806 02:09:28.380234 28756 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRTVuz1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:09:28.380303 28756 mini_ranger.cc:193] Using host: 127.28.21.20
I20260806 02:09:28.384410 28756 mini_ranger.cc:227] Ranger admin URL: http://127.28.21.20:50177
Aug 06, 2026 2:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:09:29.565181 28756 mini_ranger.cc:160] Time spent starting Ranger: real 8.484s user 0.000s sys 0.011s
Aug 06, 2026 2:09:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:09:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:09:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:09:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:09:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:09:50.339568 28756 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangerkms_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/rangerkms_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/HTTP_127.28.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:09:50 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982190, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:09:50.429777 28756 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:09:50.440054 28756 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:09:51.333887 28756 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:09:51.333984 28756 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms
I20260806 02:09:51.334008 28756 mini_ranger_kms.cc:227] Using postgres at 127.28.21.20:59495
2026-08-06 02:09:51,503 [I] DB FLAVOR :POSTGRES
2026-08-06 02:09:51,504 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:09:51,504 [I] Checking connection
2026-08-06 02:09:51,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:09:51,747 [I] connection success
2026-08-06 02:09:51,747 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:09:51,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:09:51,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:09:51,994 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:09:51,995 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:09:51,995 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:09:51,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.21.20:59495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:09:52,269 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:09:52.727324 28756 mini_ranger_kms.cc:310] Created kms service
I20260806 02:09:52.890911 28756 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:09:52.934600 28756 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:09:52 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982190, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:53.325378 28756 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:09:53.325534 28756 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRTVuz1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRTVuz1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:09:53.325556 28756 mini_ranger_kms.cc:240] Using host: 127.28.21.20
I20260806 02:09:53.329967 28756 mini_ranger_kms.cc:276] Ranger KMS PID: 9223
I20260806 02:09:53.330058 28756 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.21.20:51469
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:09:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:09:55 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785982195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:55.680009 28756 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.250s user 0.015s sys 0.140s
I20260806 02:09:55.680159 28756 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:09:55.680258 28756 mini_ranger_kms.cc:401] 127.28.21.20:51469/kms/v1/keys
Aug 06 02:09:56 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
Aug 06 02:09:56 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.21.20@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
Aug 06 02:09:58 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:09:58.743993 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32823
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:32823
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:09:58.856482 9280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:58.856751 9280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:58.856791 9280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:58.860664 9280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:09:58.860720 9280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:58.860739 9280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:58.860756 9280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:09:58.860771 9280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:09:58.866145 9280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.21.20:51469/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:32823
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:58.867256 9280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:58.868283 9280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:58.877058 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:58.877115 9280 server_base.cc:1034] running on GCE node
W20260806 02:09:58.877122 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:58.877058 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:58.877861 9280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:58.879092 9280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:58.880249 9280 hybrid_clock.cc:648] HybridClock initialized: now 1785982198880231 us; error 30 us; skew 500 ppm
Aug 06 02:09:58 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982198, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:58.883109 9280 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:09:58.883980 9280 webserver.cc:460] Webserver started at http://127.28.21.62:42221/ using document root <none> and password file <none>
I20260806 02:09:58.884518 9280 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:58.884595 9280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:58.884821 9280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:58 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982198, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:58.939615 9280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "cf321782dc054806a4456319692822f3"
format_stamp: "Formatted at 2026-08-06 02:09:58 on dist-test-slave-gp6n"
server_key: "01191093fbc3379db8957006ed236664"
server_key_iv: "69ae39006d082a5937198917bb45ad9c"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:58.940266 9280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "cf321782dc054806a4456319692822f3"
format_stamp: "Formatted at 2026-08-06 02:09:58 on dist-test-slave-gp6n"
server_key: "01191093fbc3379db8957006ed236664"
server_key_iv: "69ae39006d082a5937198917bb45ad9c"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:58.944651 9280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:09:58.962551 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:09:58.964246 9280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:09:58.964388 9280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "cf321782dc054806a4456319692822f3"
format_stamp: "Formatted at 2026-08-06 02:09:58 on dist-test-slave-gp6n"
server_key: "01191093fbc3379db8957006ed236664"
server_key_iv: "69ae39006d082a5937198917bb45ad9c"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:58.964529 9280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:09:58.980870 9280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:09:58.984402 9280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:09:58.984607 9280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:09:58.993489 9280 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:32823
I20260806 02:09:58.993559 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:32823 every 8 connection(s)
I20260806 02:09:58.994700 9280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:09:58.997542 9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:09:59.001942 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9280
I20260806 02:09:59.002064 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:09:59.002378 28756 external_mini_cluster.cc:1428] Setting key 2b333ab9d1e91db792bf5a2cc7094c4e
I20260806 02:09:59.004244 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Bootstrap starting.
I20260806 02:09:59.006592 9356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Neither blocks nor log segments found. Creating new log.
I20260806 02:09:59.007503 9356 log.cc:824] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:09:59.009532 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: No bootstrap required, opened a new log
Aug 06 02:09:59 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:09:59.013808 9356 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:09:59.014081 9356 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:09:59.014142 9356 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf321782dc054806a4456319692822f3, State: Initialized, Role: FOLLOWER
I20260806 02:09:59.014660 9356 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:09:59.014850 9356 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:09:59.014922 9356 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:09:59.015024 9356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:09:59.016252 9356 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:09:59.016636 9356 leader_election.cc:302] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf321782dc054806a4456319692822f3; no voters:
I20260806 02:09:59.017007 9356 leader_election.cc:288] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:09:59.017127 9362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:09:59.017343 9362 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 LEADER]: Becoming Leader. State: Replica: cf321782dc054806a4456319692822f3, State: Running, Role: LEADER
I20260806 02:09:59.017638 9362 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:09:59.018499 9356 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:09:59.019146 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf321782dc054806a4456319692822f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } } }
I20260806 02:09:59.019269 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:09:59.019464 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf321782dc054806a4456319692822f3. Latest consensus state: current_term: 1 leader_uuid: "cf321782dc054806a4456319692822f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } } }
I20260806 02:09:59.019539 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:09:59.019781 9368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:09:59.020954 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:09:59.003958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.28.21.62:32823)
0806 02:09:59.004507 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 02:09:59.004527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:09:59.004573 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:09:59.005451 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:09:59.005474 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:09:59.005917 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:09:59.006285 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:09:59.007297 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:59.008797 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:59.010044 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:09:59.010771 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:09:59.010939 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:09:59.014061 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:09:59.014068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:09:59.014084 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:09:59.014120 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:09:59.016414 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:59.016900 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:59.016907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:59.016917 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:59.017168 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:09:59.017631 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:09:59.017634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:09:59.017636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:09:59.018241 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:09:59.019128 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0806 02:09:59.019431 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:09:59.019877 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":144,"threads_started":1}
I20260806 02:09:59.022893 9368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:09:59.023779 9356 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:09:59.032650 9368 catalog_manager.cc:1347] Generated new cluster ID: 3a81d38c2c33465c99b39e56e5724cb6
I20260806 02:09:59.032791 9368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:09:59.059947 9368 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:09:59.061367 9368 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:09:59.070052 9368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Generated new TSK 0
I20260806 02:09:59.071130 9368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:09:59.758250 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:09:59.911005 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:09:59.911365 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:09:59.911437 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:09:59.917562 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:09:59.917642 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:09:59.917727 9403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:09:59.925546 9403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:09:59.926783 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:09:59.927879 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:09:59.939836 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:59.939978 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:09:59.939836 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:09:59.940930 9403 server_base.cc:1034] running on GCE node
I20260806 02:09:59.941354 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:09:59.942023 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:09:59.943210 9403 hybrid_clock.cc:648] HybridClock initialized: now 1785982199943143 us; error 84 us; skew 500 ppm
Aug 06 02:09:59 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:09:59.951011 9403 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:09:59.952188 9403 webserver.cc:460] Webserver started at http://127.28.21.1:38281/ using document root <none> and password file <none>
I20260806 02:09:59.952927 9403 fs_manager.cc:359] Metadata directory not provided
I20260806 02:09:59.953081 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:09:59.953356 9403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:09:59 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:09:59.971065 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "b6a4bb933e444e98b54b188611c0220a"
format_stamp: "Formatted at 2026-08-06 02:09:59 on dist-test-slave-gp6n"
server_key: "a9934454139d2a47ef15321ce939000e"
server_key_iv: "d7c821eb7b44c5c0fe2f214a74f7dace"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:59.971850 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "b6a4bb933e444e98b54b188611c0220a"
format_stamp: "Formatted at 2026-08-06 02:09:59 on dist-test-slave-gp6n"
server_key: "a9934454139d2a47ef15321ce939000e"
server_key_iv: "d7c821eb7b44c5c0fe2f214a74f7dace"
server_key_version: "kuduclusterkey@0"
I20260806 02:09:59.977556 9403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:10:00.010850 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.012631 9403 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260806 02:10:00.012820 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "b6a4bb933e444e98b54b188611c0220a"
format_stamp: "Formatted at 2026-08-06 02:09:59 on dist-test-slave-gp6n"
server_key: "a9934454139d2a47ef15321ce939000e"
server_key_iv: "d7c821eb7b44c5c0fe2f214a74f7dace"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.013022 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:00.029359 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:00.033209 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:00.033489 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:00.034200 9403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:00.035349 9403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:00.035472 9403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.035570 9403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:00.035624 9403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.046129 9403 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:33199
I20260806 02:10:00.047348 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:00.047708 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9403
I20260806 02:10:00.047820 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:10:00.048135 28756 external_mini_cluster.cc:1428] Setting key 83b96e7e39b7006dc53f1836c3132a24
I20260806 02:10:00.055280 9545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:33199 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:00.089460 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:00.065964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33761 (local address 127.28.21.62:32823)
0806 02:10:00.066359 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:00.066369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:00.072854 (+ 6485us) server_negotiation.cc:407] Connection header received
0806 02:10:00.075353 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:00.075358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:00.075428 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:00.075507 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:00.077433 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.078438 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.080392 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.080804 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.080908 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:00.084339 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:00.084343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:00.084347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:00.084379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:00.086732 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:00.087446 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:00.087450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:00.087452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:00.087552 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:00.088016 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:00.088020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:00.088022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:00.088260 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:00.088337 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:00.089091 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 02:10:00.089233 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260806 02:10:00.091243 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:00.066511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.1:33761)
0806 02:10:00.072618 (+ 6107us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:00.072695 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:00.075046 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:00.076013 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:00.076024 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:00.076429 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:00.077225 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:00.077249 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.078622 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.078626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:00.080230 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:00.080242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.081149 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.081153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:00.081378 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:00.082198 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:00.082221 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:00.084110 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:00.086891 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:00.086898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:00.086916 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:00.087302 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:00.087667 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:00.087670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:00.087672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:00.087887 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:00.088442 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:00.088449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:00.088799 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 02:10:00.089743 (+ 944us) client_negotiation.cc:240] Negotiation successful
0806 02:10:00.090110 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"thread_start_us":151,"threads_started":1}
I20260806 02:10:00.092955 9546 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:00.093223 9546 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:00.093806 9546 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:00.095633 9316 ts_manager.cc:194] Registered new tserver with Master: b6a4bb933e444e98b54b188611c0220a (127.28.21.1:33199)
I20260806 02:10:00.097174 9316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:33761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:10:00.105810 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:00.283541 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:00.283932 9552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:00.284010 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:00.290132 9552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:00.290218 9552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:00.290309 9552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:00.298154 9552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:00.299582 9552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:00.300951 9552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:00.314195 9559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:00.314669 9558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:00.315397 9561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:00.315698 9552 server_base.cc:1034] running on GCE node
I20260806 02:10:00.316256 9552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:00.317020 9552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:00.324378 9552 hybrid_clock.cc:648] HybridClock initialized: now 1785982200324293 us; error 96 us; skew 500 ppm
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:00.328215 9552 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:00.329355 9552 webserver.cc:460] Webserver started at http://127.28.21.2:36999/ using document root <none> and password file <none>
I20260806 02:10:00.330039 9552 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:00.330109 9552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:00.330322 9552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:00.351600 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "84de407b61e048a991b99b9052d5ca86"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "7a2c67479c4e33ba4e4c206fde9a58cd"
server_key_iv: "09c3e2cd3f729163ac608b8e4a45983c"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.352926 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "84de407b61e048a991b99b9052d5ca86"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "7a2c67479c4e33ba4e4c206fde9a58cd"
server_key_iv: "09c3e2cd3f729163ac608b8e4a45983c"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.358182 9552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:10:00.377333 9568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.379228 9552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:10:00.379393 9552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "84de407b61e048a991b99b9052d5ca86"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "7a2c67479c4e33ba4e4c206fde9a58cd"
server_key_iv: "09c3e2cd3f729163ac608b8e4a45983c"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.379549 9552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:00.410456 9552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:00.414265 9552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:00.414525 9552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:00.415248 9552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:00.416308 9552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:00.416385 9552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.416442 9552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:00.416476 9552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:00.430651 9552 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:39919
I20260806 02:10:00.431900 9552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:00.433403 9689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:39919 every 8 connection(s)
I20260806 02:10:00.440321 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9552
I20260806 02:10:00.440471 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:10:00.440788 28756 external_mini_cluster.cc:1428] Setting key 50064d6db664199064660a45f4b072e7
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:00.455505 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:00.437039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:43755 (local address 127.28.21.62:32823)
0806 02:10:00.437192 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:00.437198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:00.437717 (+ 519us) server_negotiation.cc:407] Connection header received
0806 02:10:00.438903 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:00.438908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:00.438970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:00.439030 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:00.440807 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.441635 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.447068 (+ 5433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.447451 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.447539 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:00.450869 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:00.450873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:00.450876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:00.450908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:00.452886 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:00.453563 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:00.453567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:00.453569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:00.453662 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:00.454094 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:00.454098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:00.454100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:00.454330 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:00.454440 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:00.455170 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:10:00.455317 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:10:00.457543 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:00.435876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.2:43755)
0806 02:10:00.437520 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:00.437583 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:00.438615 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:00.439495 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:00.439505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:00.439897 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:00.440615 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:00.440637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.445223 (+ 4586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.445227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:00.446909 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:00.446921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:00.447763 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:00.447767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:00.447997 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:00.448721 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:00.448743 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:00.450649 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:00.453046 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:00.453053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:00.453071 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:00.453434 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:00.453774 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:00.453777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:00.453779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:00.453974 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:00.454513 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:00.454520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:00.454870 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 02:10:00.456019 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0806 02:10:00.456391 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"thread_start_us":142,"threads_started":1}
I20260806 02:10:00.461243 9690 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:00.461508 9690 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:00.462134 9690 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:00.463439 9316 ts_manager.cc:194] Registered new tserver with Master: 84de407b61e048a991b99b9052d5ca86 (127.28.21.2:39919)
I20260806 02:10:00.464183 9316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:43755
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:10:00.527813 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
W20260806 02:10:00.864979 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:00.865360 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:00.865860 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:00.872192 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:00.872373 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:00.872520 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:00.880568 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:00.882207 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:00.883662 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:00.897500 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:00.897776 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:00.897471 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:00.899168 9696 server_base.cc:1034] running on GCE node
I20260806 02:10:00.899783 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:00.900628 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:00.932230 9696 hybrid_clock.cc:648] HybridClock initialized: now 1785982200932118 us; error 131 us; skew 500 ppm
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:00.936636 9696 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:00.938186 9696 webserver.cc:460] Webserver started at http://127.28.21.3:37775/ using document root <none> and password file <none>
I20260806 02:10:00.939045 9696 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:00.939206 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:00.939513 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:10:00 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:00.969256 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "a84f04580df84ed0b8e31c4b37b8b26e"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "b01c8d26cf474b0749de9feeaa2c73f5"
server_key_iv: "e242965d6a281df512e5f1858abdfea1"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.970171 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "a84f04580df84ed0b8e31c4b37b8b26e"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "b01c8d26cf474b0749de9feeaa2c73f5"
server_key_iv: "e242965d6a281df512e5f1858abdfea1"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:00.980983 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.003s
I20260806 02:10:01.014847 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.016795 9696 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260806 02:10:01.017035 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a84f04580df84ed0b8e31c4b37b8b26e"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "b01c8d26cf474b0749de9feeaa2c73f5"
server_key_iv: "e242965d6a281df512e5f1858abdfea1"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:01.017266 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:01.058753 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:01.069923 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:01.070147 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:01.070930 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:01.071931 9696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:01.071995 9696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.072062 9696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:01.072093 9696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.093962 9696 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:35159
I20260806 02:10:01.095381 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:01.096594 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9696
I20260806 02:10:01.096735 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:10:01.097043 28756 external_mini_cluster.cc:1428] Setting key 9a36a70ce56d612d63f4b5c4800659df
I20260806 02:10:01.100072 9546 heartbeater.cc:503] Master 127.28.21.62:32823 was elected leader, sending a full tablet report...
I20260806 02:10:01.121078 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:35159 every 8 connection(s)
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:01.155102 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:01.117057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.3:55219)
0806 02:10:01.130837 (+ 13780us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:01.130925 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:01.132176 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:01.132854 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:01.132865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:01.133297 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:01.134041 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:01.134062 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.135354 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.135358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:01.136924 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:01.136936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.137630 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.137634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:01.137858 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:01.146646 (+ 8788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:01.146675 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:01.148705 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:01.151096 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:01.151105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:01.151129 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:01.151660 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:01.152047 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:01.152051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:01.152053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:01.152275 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:01.152844 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:01.152853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:01.153238 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 02:10:01.153490 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 02:10:01.153882 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13451,"thread_start_us":160,"threads_started":1}
I20260806 02:10:01.156095 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:01.116387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:55219 (local address 127.28.21.62:32823)
0806 02:10:01.121527 (+ 5140us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:01.121532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:01.131069 (+ 9537us) server_negotiation.cc:407] Connection header received
0806 02:10:01.132456 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:01.132460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:01.132517 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:01.132647 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:01.134344 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.135214 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.137063 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.137445 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.137545 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:01.148911 (+ 11366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:01.148915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:01.148918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:01.148951 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:01.150940 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:01.151811 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:01.151816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:01.151818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:01.151932 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:01.152417 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:01.152420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:01.152422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:01.152682 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:01.152781 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:01.155751 (+ 2970us) server_negotiation.cc:299] Negotiation successful
0806 02:10:01.155911 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5050,"thread_start_us":74,"threads_started":1}
I20260806 02:10:01.157946 9831 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:01.158263 9831 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:01.158962 9831 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:01.160226 9316 ts_manager.cc:194] Registered new tserver with Master: a84f04580df84ed0b8e31c4b37b8b26e (127.28.21.3:35159)
I20260806 02:10:01.160975 9316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:55219
I20260806 02:10:01.171728 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:01.172559 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9403
I20260806 02:10:01.193857 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9552
I20260806 02:10:01.203773 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9696
I20260806 02:10:01.227474 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9280
I20260806 02:10:01.238935 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32823
--webserver_interface=127.28.21.62
--webserver_port=42221
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:32823
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:01.354707 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:01.354984 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:01.355023 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:01.359117 9835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:01.359180 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:01.359200 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:01.359216 9835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:01.359232 9835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:01.364791 9835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:32823
--ranger_config_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=42221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:01.365814 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:01.366914 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:01.375579 9841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:01.375573 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:01.375568 9840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:01.375609 9835 server_base.cc:1034] running on GCE node
I20260806 02:10:01.376319 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:01.377476 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:01.378666 9835 hybrid_clock.cc:648] HybridClock initialized: now 1785982201378637 us; error 50 us; skew 500 ppm
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982201, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:01.381716 9835 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:01.382727 9835 webserver.cc:460] Webserver started at http://127.28.21.62:42221/ using document root <none> and password file <none>
I20260806 02:10:01.383277 9835 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:01.383363 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982201, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:01.469261 9835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:10:01.472338 9852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.474023 9835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:10:01.474188 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "cf321782dc054806a4456319692822f3"
format_stamp: "Formatted at 2026-08-06 02:09:58 on dist-test-slave-gp6n"
server_key: "01191093fbc3379db8957006ed236664"
server_key_iv: "69ae39006d082a5937198917bb45ad9c"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:01.475018 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:01.499387 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:01.503005 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:01.503182 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:01.511154 9835 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:32823
I20260806 02:10:01.511161 9910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:32823 every 8 connection(s)
I20260806 02:10:01.512364 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:01.516788 9911 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:10:01.518394 9911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Bootstrap starting.
I20260806 02:10:01.518957 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9835
I20260806 02:10:01.519475 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:33199
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:32823
--webserver_port=38281
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:01.536576 9911 log.cc:824] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:01.543541 9911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:01.543956 9911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Bootstrap complete.
I20260806 02:10:01.548282 9911 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:10:01.548791 9911 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf321782dc054806a4456319692822f3, State: Initialized, Role: FOLLOWER
I20260806 02:10:01.549329 9911 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:10:01.549468 9911 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:01.549556 9911 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:01.549665 9911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:10:01.551622 9911 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:10:01.551947 9911 leader_election.cc:302] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf321782dc054806a4456319692822f3; no voters:
I20260806 02:10:01.552263 9911 leader_election.cc:288] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:10:01.552413 9915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:10:01.552663 9915 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [term 2 LEADER]: Becoming Leader. State: Replica: cf321782dc054806a4456319692822f3, State: Running, Role: LEADER
I20260806 02:10:01.553066 9915 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } }
I20260806 02:10:01.553447 9911 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:01.554742 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf321782dc054806a4456319692822f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } } }
I20260806 02:10:01.554879 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:01.555626 9917 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf321782dc054806a4456319692822f3. Latest consensus state: current_term: 2 leader_uuid: "cf321782dc054806a4456319692822f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf321782dc054806a4456319692822f3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32823 } } }
I20260806 02:10:01.555768 9917 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:01.555804 9911 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:10:01.556166 9925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:01.562623 9925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:01.563580 9925 catalog_manager.cc:1259] Loaded cluster ID: 3a81d38c2c33465c99b39e56e5724cb6
I20260806 02:10:01.563647 9925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:01.566715 9925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:01.567531 9925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Loaded TSK: 0
I20260806 02:10:01.569044 9925 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:10:01.671857 9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:01.672366 9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:01.672508 9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:01.679476 9913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:01.679577 9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:01.679678 9913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:01.688505 9913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:33199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=38281
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:01.690186 9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:01.691702 9913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:01.704814 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:01.705168 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:01.705274 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:01.705649 9913 server_base.cc:1034] running on GCE node
I20260806 02:10:01.706142 9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:01.707139 9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:01.708362 9913 hybrid_clock.cc:648] HybridClock initialized: now 1785982201708318 us; error 49 us; skew 500 ppm
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982201, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:01.712188 9913 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:01.713354 9913 webserver.cc:460] Webserver started at http://127.28.21.1:38281/ using document root <none> and password file <none>
I20260806 02:10:01.714097 9913 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:01.714203 9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982201, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:01.777004 9913 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.000s
I20260806 02:10:01.783525 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.787860 9913 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260806 02:10:01.788053 9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "b6a4bb933e444e98b54b188611c0220a"
format_stamp: "Formatted at 2026-08-06 02:09:59 on dist-test-slave-gp6n"
server_key: "a9934454139d2a47ef15321ce939000e"
server_key_iv: "d7c821eb7b44c5c0fe2f214a74f7dace"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:01.788812 9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:01.820739 9913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:01.824801 9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:01.825045 9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:01.825798 9913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:01.826961 9913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:01.827031 9913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.827107 9913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:01.827138 9913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:01.839915 9913 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:33199
I20260806 02:10:01.839951 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:33199 every 8 connection(s)
I20260806 02:10:01.841284 9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:01.849640 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 9913
I20260806 02:10:01.850214 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:39919
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:32823
--webserver_port=36999
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:01 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982201, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:01.866329 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:01.845234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.1:50139)
0806 02:10:01.847867 (+ 2633us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:01.847936 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:01.849152 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:01.851083 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:01.851092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:01.851467 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:01.852121 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:01.852139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.853774 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.853778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:01.855244 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:01.855255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.856664 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.856669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:01.856835 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:01.857523 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:01.857542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:01.859359 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:01.862394 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:01.862428 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:01.862452 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:01.862749 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:01.863145 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:01.863149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:01.863151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:01.863306 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:01.864176 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:01.864184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:01.864564 (+ 380us) client_negotiation.cc:769] Sending connection context
0806 02:10:01.864797 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 02:10:01.865177 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":181,"threads_started":1}
I20260806 02:10:01.868228 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:01.846949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:50139 (local address 127.28.21.62:32823)
0806 02:10:01.849770 (+ 2821us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:01.849799 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:01.849851 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:10:01.850125 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:01.850137 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:01.850614 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:01.850997 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:01.852360 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.853570 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.855427 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:01.856243 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:01.856416 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:01.859654 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:01.859661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:01.859680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:01.859716 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:01.862253 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:01.862898 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:01.862905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:01.862910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:01.863038 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:01.863436 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:01.863439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:01.863441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:01.863958 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:01.864110 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:01.866918 (+ 2808us) server_negotiation.cc:299] Negotiation successful
0806 02:10:01.867239 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2628,"thread_start_us":134,"threads_started":1}
I20260806 02:10:01.872092 10083 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:01.872357 10083 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:01.872979 10083 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
W20260806 02:10:01.875517 10083 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:32823 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:10:01.875662 10083 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:01.875873 10083 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:01.876922 10083 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:01.877110 10083 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
W20260806 02:10:01.877990 10083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:10:01.996165 10089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:01.996487 10089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:01.996565 10089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:02.002009 10089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:02.002197 10089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:02.002352 10089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:02.010448 10089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:39919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=36999
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:02.011826 10089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:02.012939 10089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:02.024226 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:02.024430 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:02.024519 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:02.024927 10089 server_base.cc:1034] running on GCE node
I20260806 02:10:02.025702 10089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:02.026681 10089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:02.030006 10089 hybrid_clock.cc:648] HybridClock initialized: now 1785982202029966 us; error 49 us; skew 500 ppm
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:02.034309 10089 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:02.035684 10089 webserver.cc:460] Webserver started at http://127.28.21.2:36999/ using document root <none> and password file <none>
I20260806 02:10:02.036550 10089 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:02.036684 10089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:02.063956 10089 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260806 02:10:02.070840 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.072695 10089 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260806 02:10:02.072855 10089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "84de407b61e048a991b99b9052d5ca86"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "7a2c67479c4e33ba4e4c206fde9a58cd"
server_key_iv: "09c3e2cd3f729163ac608b8e4a45983c"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:02.073630 10089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:02.100185 10089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:02.104322 10089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:02.104553 10089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:02.105377 10089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:02.106549 10089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:02.106619 10089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.106678 10089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:02.106715 10089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.118285 10089 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:39919
I20260806 02:10:02.118438 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:39919 every 8 connection(s)
I20260806 02:10:02.119552 10089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:02.120217 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10089
I20260806 02:10:02.120707 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.21.20:51469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:35159
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:32823
--webserver_port=37775
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:02.140612 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:02.123363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:53019 (local address 127.28.21.62:32823)
0806 02:10:02.123524 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:02.123529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:02.124614 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 02:10:02.125971 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:02.125975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:02.126040 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:02.126143 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:02.127809 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.128616 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.130485 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.130814 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.130936 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:02.134729 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:02.134736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:02.134741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:02.134782 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:02.136898 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.138326 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.138331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.138333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.138454 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.138929 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.138935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.138940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.139237 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:02.139340 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:02.140219 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 02:10:02.140416 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:10:02.143010 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:02.122229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.2:53019)
0806 02:10:02.124413 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:02.124475 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:02.125700 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:02.126369 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:02.126381 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:02.126792 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:02.127610 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:02.127637 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.128752 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.128756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:02.130239 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:02.130251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.131017 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.131025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:02.131285 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:02.132243 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:02.132277 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:02.134380 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:02.137096 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:02.137105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:02.137129 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:02.138189 (+ 1060us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:02.138573 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:02.138576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:02.138578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:02.138798 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:02.139427 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:02.139437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:02.139859 (+ 422us) client_negotiation.cc:769] Sending connection context
0806 02:10:02.141206 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0806 02:10:02.141606 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"thread_start_us":176,"threads_started":1}
I20260806 02:10:02.144407 10228 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:02.144685 10228 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:02.145335 10228 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
W20260806 02:10:02.147487 10228 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:32823 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:10:02.147778 10228 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:02.148140 10228 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:02.150843 10228 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:02.151057 10228 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
W20260806 02:10:02.151935 10228 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:10:02.273380 10232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:02.273707 10232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:02.273792 10232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:02.278193 10232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:02.278291 10232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:02.278388 10232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:02.285773 10232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.21.20:51469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:35159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=37775
--tserver_master_addrs=127.28.21.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:02.287266 10232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:02.288645 10232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:02.300323 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:02.300285 10241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:02.300644 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:02.301334 10232 server_base.cc:1034] running on GCE node
I20260806 02:10:02.301858 10232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:02.302760 10232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:02.303975 10232 hybrid_clock.cc:648] HybridClock initialized: now 1785982202303935 us; error 48 us; skew 500 ppm
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:02.307729 10232 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:02.308915 10232 webserver.cc:460] Webserver started at http://127.28.21.3:37775/ using document root <none> and password file <none>
I20260806 02:10:02.309692 10232 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:02.309808 10232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:02.335398 10232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:10:02.338104 10250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.339589 10232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:02.339741 10232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a84f04580df84ed0b8e31c4b37b8b26e"
format_stamp: "Formatted at 2026-08-06 02:10:00 on dist-test-slave-gp6n"
server_key: "b01c8d26cf474b0749de9feeaa2c73f5"
server_key_iv: "e242965d6a281df512e5f1858abdfea1"
server_key_version: "kuduclusterkey@0"
I20260806 02:10:02.340338 10232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:02.352738 10232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:02.356550 10232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:02.356783 10232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:02.357477 10232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:02.358642 10232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:02.358788 10232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.358865 10232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:02.358901 10232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:02.370332 10232 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:35159
I20260806 02:10:02.370429 10379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:35159 every 8 connection(s)
I20260806 02:10:02.371441 10232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:02.376735 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10232
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:02.387822 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:02.374864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:44611 (local address 127.28.21.62:32823)
0806 02:10:02.374992 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:02.374996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:02.375011 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:10:02.375780 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:02.375782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:02.375822 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:02.375906 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:02.377546 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.378234 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.379967 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.380194 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.380345 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:02.383431 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:02.383436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:02.383441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:02.383475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:02.385203 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.385856 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.385860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.385864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.385963 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.386424 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.386428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.386430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.386663 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:02.386759 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:02.387516 (+ 757us) server_negotiation.cc:299] Negotiation successful
0806 02:10:02.387647 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:10:02.388505 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:02.377793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.28.21.62:32823)
0806 02:10:02.380559 (+ 2766us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:02.380565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:02.380574 (+ 9us) server_negotiation.cc:407] Connection header received
0806 02:10:02.380601 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:02.380603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:02.380634 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:02.380732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:02.381797 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.382320 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.383312 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.383534 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.383640 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:02.385122 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:02.385131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:02.385134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:02.385157 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:02.386620 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.387043 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.387051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.387052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.387124 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:02.387415 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:02.387419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:02.387420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:02.387620 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:02.387813 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:02.388217 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:10:02.388344 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2714,"thread_start_us":113,"threads_started":1}
I20260806 02:10:02.389972 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:02.373871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32823 (local address 127.28.21.3:44611)
0806 02:10:02.374443 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:02.374503 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:02.375561 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:02.376121 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:02.376131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:02.376546 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:02.377339 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:02.377363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.378367 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.378371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:02.379833 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:02.379845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:02.380399 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:02.380404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:02.380628 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:02.381391 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:02.381413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:02.383213 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:02.385345 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:02.385352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:02.385370 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:02.385733 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:02.386069 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:02.386071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:02.386073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:02.386270 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:02.386834 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:02.386841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:02.387216 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 02:10:02.388087 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 02:10:02.388455 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260806 02:10:02.391193 10380 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32823
I20260806 02:10:02.391445 10380 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:02.391968 10380 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:02.393378 9871 ts_manager.cc:194] Registered new tserver with Master: a84f04580df84ed0b8e31c4b37b8b26e (127.28.21.3:35159)
I20260806 02:10:02.394729 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:44611
Aug 06 02:10:02 dist-test-slave-gp6n krb5kdc[8807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:02.878245 10083 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:02.878676 10083 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:02.879549 9871 ts_manager.cc:194] Registered new tserver with Master: b6a4bb933e444e98b54b188611c0220a (127.28.21.1:33199)
I20260806 02:10:02.880381 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:50139
Aug 06 02:10:03 dist-test-slave-gp6n krb5kdc[8807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982202, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for HTTP/127.28.21.20@KRBTEST.COM
I20260806 02:10:03.152170 10228 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:03.152544 10228 heartbeater.cc:511] Master 127.28.21.62:32823 requested a full tablet report, sending...
I20260806 02:10:03.153439 9871 ts_manager.cc:194] Registered new tserver with Master: 84de407b61e048a991b99b9052d5ca86 (127.28.21.2:39919)
I20260806 02:10:03.154124 9871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:53019
I20260806 02:10:03.158893 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:03.169181 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.160519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.28.21.62:32823)
0806 02:10:03.160821 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.160827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.160896 (+ 69us) server_negotiation.cc:407] Connection header received
0806 02:10:03.161057 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.161060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.161115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.161203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:03.161896 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.162520 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.163344 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.163594 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.163698 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.165172 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:03.165175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:03.165178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:03.165204 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:03.166881 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:03.167334 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:03.167337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:03.167339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:03.167447 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:03.167797 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:03.167801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:03.167803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:03.168046 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:03.168184 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.168844 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.168992 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260806 02:10:03.179200 9871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:10:03.184345 9871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:10:03.213802 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.201578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:39919 (local address 127.0.0.1:59994)
0806 02:10:03.206782 (+ 5204us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.206793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:03.206922 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:03.207725 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:03.207734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:03.207759 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:03.207937 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.207946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.209459 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.209462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.211781 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.211793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.213182 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.213186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.213314 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.213357 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:10:03.213489 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:10:03.213625 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5150,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260806 02:10:03.215240 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.201749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.28.21.2:39919)
0806 02:10:03.206646 (+ 4897us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.206653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.206893 (+ 240us) server_negotiation.cc:407] Connection header received
0806 02:10:03.207071 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.207079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.207332 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.207501 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:03.208096 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.209317 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.211932 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.212839 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.212950 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.213116 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.214173 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.214286 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4773,"thread_start_us":79,"threads_started":1}
I20260806 02:10:03.218827 10158 tablet_service.cc:1467] Processing CreateTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c (DEFAULT_TABLE table=test-table [id=480586b97d6740eba4635cf1dffc2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:03.220811 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.199813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:33199 (local address 127.0.0.1:50460)
0806 02:10:03.202151 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.202205 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:03.202339 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:03.207755 (+ 5416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:03.207758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:03.207769 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:03.214246 (+ 6477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.214254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.216596 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.216600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.217709 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.217720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.220358 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.220361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.220452 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.220474 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:10:03.220551 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:10:03.220643 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"mutex_wait_us":6323,"thread_start_us":105,"threads_started":1}
I20260806 02:10:03.221076 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1010ff35343c435cbf1f5ed0ec6ae00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:03.222791 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.199943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.28.21.1:33199)
0806 02:10:03.207207 (+ 7264us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.207214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.207237 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:10:03.207307 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.207313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.207475 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.207598 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:03.214426 (+ 6828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.216454 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.217863 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.220021 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.220132 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.220296 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.222512 (+ 2216us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.222633 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7154,"thread_start_us":84,"threads_started":1}
I20260806 02:10:03.228544 10012 tablet_service.cc:1467] Processing CreateTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c (DEFAULT_TABLE table=test-table [id=480586b97d6740eba4635cf1dffc2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:03.230005 10012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1010ff35343c435cbf1f5ed0ec6ae00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:03.233589 10407 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Bootstrap starting.
I20260806 02:10:03.237385 10407 tablet_bootstrap.cc:654] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:03.238382 10407 log.cc:824] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:03.241385 10407 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: No bootstrap required, opened a new log
I20260806 02:10:03.241621 10407 ts_tablet_manager.cc:1397] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260806 02:10:03.242254 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.200937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:35159 (local address 127.0.0.1:58338)
0806 02:10:03.202613 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.202626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:03.202707 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:03.223154 (+ 20447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:03.223158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:03.223183 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:03.223320 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.223327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.237735 (+ 14408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.237740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.239277 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.239296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.241643 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.241649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.241884 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.241920 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:10:03.241989 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:10:03.242084 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"thread_start_us":97,"threads_started":1}
I20260806 02:10:03.243269 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.201109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.28.21.3:35159)
0806 02:10:03.221531 (+ 20422us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.221538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.221566 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:10:03.221646 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.221652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.222279 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.223508 (+ 1229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:03.223684 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.224914 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.239519 (+ 14605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.240473 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.242727 (+ 2254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.242934 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.242999 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.243116 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20300,"thread_start_us":118,"threads_started":1}
I20260806 02:10:03.247310 10407 raft_consensus.cc:348] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.247145 10307 tablet_service.cc:1467] Processing CreateTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c (DEFAULT_TABLE table=test-table [id=480586b97d6740eba4635cf1dffc2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:03.247646 10407 raft_consensus.cc:374] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:03.247714 10407 raft_consensus.cc:729] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84de407b61e048a991b99b9052d5ca86, State: Initialized, Role: FOLLOWER
I20260806 02:10:03.248319 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1010ff35343c435cbf1f5ed0ec6ae00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:03.248261 10407 consensus_queue.cc:260] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.249537 10407 ts_tablet_manager.cc:1428] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260806 02:10:03.249991 10228 heartbeater.cc:503] Master 127.28.21.62:32823 was elected leader, sending a full tablet report...
I20260806 02:10:03.255992 10409 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Bootstrap starting.
I20260806 02:10:03.259647 10409 tablet_bootstrap.cc:654] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:03.260612 10409 log.cc:824] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:03.263410 10409 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: No bootstrap required, opened a new log
I20260806 02:10:03.263634 10409 ts_tablet_manager.cc:1397] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:10:03.266870 10411 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Bootstrap starting.
I20260806 02:10:03.267736 10409 raft_consensus.cc:348] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.268042 10409 raft_consensus.cc:374] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:03.268108 10409 raft_consensus.cc:729] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4bb933e444e98b54b188611c0220a, State: Initialized, Role: FOLLOWER
I20260806 02:10:03.268656 10409 consensus_queue.cc:260] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.269523 10411 tablet_bootstrap.cc:654] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:03.269799 10409 ts_tablet_manager.cc:1428] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:10:03.270188 10083 heartbeater.cc:503] Master 127.28.21.62:32823 was elected leader, sending a full tablet report...
I20260806 02:10:03.270505 10411 log.cc:824] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:03.278162 10411 tablet_bootstrap.cc:492] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: No bootstrap required, opened a new log
I20260806 02:10:03.278439 10411 ts_tablet_manager.cc:1397] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 02:10:03.283154 10411 raft_consensus.cc:348] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.283461 10411 raft_consensus.cc:374] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:03.283526 10411 raft_consensus.cc:729] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84f04580df84ed0b8e31c4b37b8b26e, State: Initialized, Role: FOLLOWER
I20260806 02:10:03.284034 10411 consensus_queue.cc:260] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.284818 10380 heartbeater.cc:503] Master 127.28.21.62:32823 was elected leader, sending a full tablet report...
I20260806 02:10:03.285208 10411 ts_tablet_manager.cc:1428] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:10:03.334345 10416 raft_consensus.cc:482] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:03.334744 10416 raft_consensus.cc:504] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.337586 10416 leader_election.cc:288] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a4bb933e444e98b54b188611c0220a (127.28.21.1:33199), 84de407b61e048a991b99b9052d5ca86 (127.28.21.2:39919)
I20260806 02:10:03.342617 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.336949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:54961 (local address 127.28.21.1:33199)
0806 02:10:03.337090 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.337095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.338017 (+ 922us) server_negotiation.cc:407] Connection header received
0806 02:10:03.338234 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.338237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.338288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.338337 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:03.338830 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.339628 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.341289 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.341883 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.341971 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.342040 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.342369 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.342471 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:10:03.343268 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.336494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:33199 (local address 127.28.21.3:54961)
0806 02:10:03.337909 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.337924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:03.338104 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:03.338530 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:03.338533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:03.338554 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:03.338703 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.338709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.339789 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.339792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.341139 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.341150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.342137 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.342140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.342242 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.342264 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:10:03.343034 (+ 770us) client_negotiation.cc:240] Negotiation successful
0806 02:10:03.343120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":137,"threads_started":1}
W20260806 02:10:03.343950 10085 tablet.cc:2369] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:03.344156 10035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1010ff35343c435cbf1f5ed0ec6ae00c" candidate_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4bb933e444e98b54b188611c0220a" is_pre_election: true
I20260806 02:10:03.344449 10035 raft_consensus.cc:2454] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84f04580df84ed0b8e31c4b37b8b26e in term 0.
I20260806 02:10:03.345127 10252 leader_election.cc:302] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84f04580df84ed0b8e31c4b37b8b26e, b6a4bb933e444e98b54b188611c0220a; no voters:
I20260806 02:10:03.345427 10416 raft_consensus.cc:2781] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:03.345496 10416 raft_consensus.cc:482] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:03.345535 10416 raft_consensus.cc:3037] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:03.346894 10416 raft_consensus.cc:504] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.347273 10416 leader_election.cc:288] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [CANDIDATE]: Term 1 election: Requested vote from peers b6a4bb933e444e98b54b188611c0220a (127.28.21.1:33199), 84de407b61e048a991b99b9052d5ca86 (127.28.21.2:39919)
I20260806 02:10:03.347997 10035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1010ff35343c435cbf1f5ed0ec6ae00c" candidate_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4bb933e444e98b54b188611c0220a"
I20260806 02:10:03.348151 10035 raft_consensus.cc:3037] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:03.349540 10035 raft_consensus.cc:2454] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84f04580df84ed0b8e31c4b37b8b26e in term 1.
I20260806 02:10:03.351104 10252 leader_election.cc:302] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84f04580df84ed0b8e31c4b37b8b26e, b6a4bb933e444e98b54b188611c0220a; no voters:
I20260806 02:10:03.351353 10416 raft_consensus.cc:2781] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:03.351586 10416 raft_consensus.cc:686] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 1 LEADER]: Becoming Leader. State: Replica: a84f04580df84ed0b8e31c4b37b8b26e, State: Running, Role: LEADER
I20260806 02:10:03.352221 10416 consensus_queue.cc:237] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } }
I20260806 02:10:03.355875 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.337453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:33065 (local address 127.28.21.2:39919)
0806 02:10:03.337596 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.337603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.343973 (+ 6370us) server_negotiation.cc:407] Connection header received
0806 02:10:03.344196 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.344200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.344257 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.344306 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:03.344776 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.345575 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.354428 (+ 8853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.355057 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.355169 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.355239 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.355631 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.355721 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:10:03.356313 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.337349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:39919 (local address 127.28.21.3:33065)
0806 02:10:03.343816 (+ 6467us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.343844 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:03.344061 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:03.344454 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:03.344459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:03.344476 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:03.344588 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.344594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.345805 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.345809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.354190 (+ 8381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:03.354207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.355373 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.355376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:03.355488 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.355511 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:10:03.356083 (+ 572us) client_negotiation.cc:240] Negotiation successful
0806 02:10:03.356173 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6378,"thread_start_us":109,"threads_started":1}
I20260806 02:10:03.356048 9870 catalog_manager.cc:5495] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e reported cstate change: term changed from 0 to 1, leader changed from <none> to a84f04580df84ed0b8e31c4b37b8b26e (127.28.21.3). New cstate: current_term: 1 leader_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 } health_report { overall_health: UNKNOWN } } }
I20260806 02:10:03.356973 10180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1010ff35343c435cbf1f5ed0ec6ae00c" candidate_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84de407b61e048a991b99b9052d5ca86" is_pre_election: true
I20260806 02:10:03.357293 10180 raft_consensus.cc:2454] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84f04580df84ed0b8e31c4b37b8b26e in term 0.
I20260806 02:10:03.357657 10179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1010ff35343c435cbf1f5ed0ec6ae00c" candidate_uuid: "a84f04580df84ed0b8e31c4b37b8b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84de407b61e048a991b99b9052d5ca86"
I20260806 02:10:03.357777 10179 raft_consensus.cc:3037] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:03.359694 10179 raft_consensus.cc:2454] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84f04580df84ed0b8e31c4b37b8b26e in term 1.
W20260806 02:10:03.372561 10231 tablet.cc:2369] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:10:03.373360 10381 tablet.cc:2369] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:03.402383 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.397080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.28.21.3:35159)
0806 02:10:03.397248 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:03.397253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:03.397271 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:10:03.397384 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:03.397390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:03.397480 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:03.397645 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:03.398092 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.399005 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.400467 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:03.400791 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:03.400901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:03.401311 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:03.401906 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:03.402049 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:03.402168 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:10:03.402235 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:10:03.410301 10035 raft_consensus.cc:1261] T 1010ff35343c435cbf1f5ed0ec6ae00c P b6a4bb933e444e98b54b188611c0220a [term 1 FOLLOWER]: Refusing update from remote peer a84f04580df84ed0b8e31c4b37b8b26e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:03.410269 10179 raft_consensus.cc:1261] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 1 FOLLOWER]: Refusing update from remote peer a84f04580df84ed0b8e31c4b37b8b26e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:03.411134 10419 consensus_queue.cc:1035] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a4bb933e444e98b54b188611c0220a" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:03.411367 10416 consensus_queue.cc:1035] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84de407b61e048a991b99b9052d5ca86" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:03.423974 10422 mvcc.cc:205] Tried to move back new op lower bound from 7315383105159098368 to 7315383104934973440. Current Snapshot: MvccSnapshot[applied={T|T < 7315383105159098368}]
I20260806 02:10:03.428903 10424 mvcc.cc:205] Tried to move back new op lower bound from 7315383105159098368 to 7315383104934973440. Current Snapshot: MvccSnapshot[applied={T|T < 7315383105159098368}]
I20260806 02:10:03.440230 9870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:03.440469 9870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:03.443683 9870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf321782dc054806a4456319692822f3: Sending DeleteTablet for 3 replicas of tablet 1010ff35343c435cbf1f5ed0ec6ae00c
I20260806 02:10:03.445086 10012 tablet_service.cc:1514] Processing DeleteTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:03 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:50460
I20260806 02:10:03.445256 10307 tablet_service.cc:1514] Processing DeleteTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:03 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:58338
I20260806 02:10:03.446118 10158 tablet_service.cc:1514] Processing DeleteTablet for tablet 1010ff35343c435cbf1f5ed0ec6ae00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:03 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:59994
I20260806 02:10:03.447569 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9913
I20260806 02:10:03.452081 10434 tablet_replica.cc:330] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: stopping tablet replica
I20260806 02:10:03.452518 10434 raft_consensus.cc:2229] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:03.452947 10434 raft_consensus.cc:2258] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:03.456259 10434 ts_tablet_manager.cc:1905] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:03.459681 10434 ts_tablet_manager.cc:1918] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:03.459807 10434 log.cc:1196] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/1010ff35343c435cbf1f5ed0ec6ae00c
I20260806 02:10:03.460168 10434 ts_tablet_manager.cc:1939] T 1010ff35343c435cbf1f5ed0ec6ae00c P 84de407b61e048a991b99b9052d5ca86: Deleting consensus metadata
I20260806 02:10:03.461647 9856 catalog_manager.cc:4841] TS 84de407b61e048a991b99b9052d5ca86 (127.28.21.2:39919): tablet 1010ff35343c435cbf1f5ed0ec6ae00c (table test-table [id=480586b97d6740eba4635cf1dffc2902]) successfully deleted
I20260806 02:10:03.462100 10433 tablet_replica.cc:330] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: stopping tablet replica
I20260806 02:10:03.467147 10433 raft_consensus.cc:2229] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:10:03.467681 10433 raft_consensus.cc:2258] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:03.469491 10433 ts_tablet_manager.cc:1905] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:03.472545 10433 ts_tablet_manager.cc:1918] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:03.472663 10433 log.cc:1196] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/1010ff35343c435cbf1f5ed0ec6ae00c
I20260806 02:10:03.473014 10433 ts_tablet_manager.cc:1939] T 1010ff35343c435cbf1f5ed0ec6ae00c P a84f04580df84ed0b8e31c4b37b8b26e: Deleting consensus metadata
I20260806 02:10:03.474395 9855 catalog_manager.cc:4841] TS a84f04580df84ed0b8e31c4b37b8b26e (127.28.21.3:35159): tablet 1010ff35343c435cbf1f5ed0ec6ae00c (table test-table [id=480586b97d6740eba4635cf1dffc2902]) successfully deleted
W20260806 02:10:03.474797 9854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:33199 (error 108)
I20260806 02:10:03.475857 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10089
I20260806 02:10:03.486713 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:03.486366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:33199 (local address 127.0.0.1:50462)
0806 02:10:03.486518 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:03.486608 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 02:10:03.493073 9854 catalog_manager.cc:4584] TS b6a4bb933e444e98b54b188611c0220a (127.28.21.1:33199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1010ff35343c435cbf1f5ed0ec6ae00c: Network error: Client connection negotiation failed: client connection to 127.28.21.1:33199: connect: Connection refused (error 111)
I20260806 02:10:03.494423 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10232
I20260806 02:10:03.505581 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 9835
I20260806 02:10:03.514021 28756 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:10:03.859232 28756 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:10:03.859370 28756 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:10:04.204171 28756 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:10:04.204 UTC [8837] LOG: received smart shutdown request
2026-08-06 02:10:04.208 UTC [8837] LOG: background worker "logical replication launcher" (PID 8846) exited with exit code 1
2026-08-06 02:10:04.209 UTC [8841] LOG: shutting down
2026-08-06 02:10:04.391 UTC [8837] LOG: database system is shut down
2026-08-06T02:10:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48343 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:10:04 dist-test-slave-gp6n krb5kdc[10445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:10:04 dist-test-slave-gp6n krb5kdc[10445](info): set up 2 sockets
Aug 06 02:10:04 dist-test-slave-gp6n krb5kdc[10445](info): commencing operation
krb5kdc: starting...
W20260806 02:10:05.620925 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:10:05 dist-test-slave-gp6n krb5kdc[10445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:10:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:10:05.743518 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41353
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:05.855239 10458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:05.855499 10458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:05.855540 10458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:05.859541 10458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:05.859593 10458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:05.859614 10458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:05.859632 10458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:05.859647 10458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:05.864811 10458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:41353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:05.865784 10458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:05.866767 10458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:05.875147 10465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:05.875159 10467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:05.875159 10464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:05.875233 10458 server_base.cc:1034] running on GCE node
I20260806 02:10:05.876031 10458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:05.877023 10458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:05.878178 10458 hybrid_clock.cc:648] HybridClock initialized: now 1785982205878151 us; error 42 us; skew 500 ppm
Aug 06 02:10:05 dist-test-slave-gp6n krb5kdc[10445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:05.884714 10458 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:05.885670 10458 webserver.cc:460] Webserver started at http://127.28.21.62:41719/ using document root <none> and password file <none>
I20260806 02:10:05.886253 10458 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:05.886345 10458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:05.886608 10458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:05.888643 10458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "34a14863bc8243b49329e208df5b72f8"
format_stamp: "Formatted at 2026-08-06 02:10:05 on dist-test-slave-gp6n"
server_key: "95785bff3e9a851ed07dcea415b9878a"
server_key_iv: "4e83afaa36578a4df8012e20c757200f"
server_key_version: "encryptionkey@0"
I20260806 02:10:05.889215 10458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "34a14863bc8243b49329e208df5b72f8"
format_stamp: "Formatted at 2026-08-06 02:10:05 on dist-test-slave-gp6n"
server_key: "95785bff3e9a851ed07dcea415b9878a"
server_key_iv: "4e83afaa36578a4df8012e20c757200f"
server_key_version: "encryptionkey@0"
I20260806 02:10:05.893011 10458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:10:05.895412 10474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:05.896610 10458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:05.896761 10458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "34a14863bc8243b49329e208df5b72f8"
format_stamp: "Formatted at 2026-08-06 02:10:05 on dist-test-slave-gp6n"
server_key: "95785bff3e9a851ed07dcea415b9878a"
server_key_iv: "4e83afaa36578a4df8012e20c757200f"
server_key_version: "encryptionkey@0"
I20260806 02:10:05.896872 10458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:05.910804 10458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:05.914059 10458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:05.914265 10458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:05.921260 10458 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:41353
I20260806 02:10:05.921281 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:41353 every 8 connection(s)
I20260806 02:10:05.922197 10458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:05.924911 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:05.928494 10529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:05.929970 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10458
I20260806 02:10:05.930102 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:10:05.930478 28756 external_mini_cluster.cc:1428] Setting key bf5271d514b0af34fa57e48e3f93ada0
Aug 06 02:10:05 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.61@KRBTEST.COM" created.
I20260806 02:10:05.943668 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.930778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.28.21.62:41353)
0806 02:10:05.931332 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:05.931346 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:05.931418 (+ 72us) server_negotiation.cc:407] Connection header received
0806 02:10:05.932444 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:05.932451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:05.932627 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:05.932809 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:05.933675 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:05.934608 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:05.935612 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:05.935969 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:05.936132 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:05.938790 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:05.938796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:05.938805 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:05.938825 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:05.940487 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:05.940931 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:05.940935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:05.940936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:05.941089 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:05.941450 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:05.941454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:05.941456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:05.941877 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:05.942042 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:05.942379 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:10:05.942651 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":134,"threads_started":1}
I20260806 02:10:05.943668 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.930777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41353 (local address 127.0.0.1:49078)
0806 02:10:05.931267 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:05.931316 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:05.932150 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:05.932810 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:05.932815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:05.933258 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:05.933526 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:05.933540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:05.934739 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:05.934741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:05.935496 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:05.935504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:05.936117 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:05.936121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:05.936236 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:05.936860 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:05.936876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:05.938647 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:05.940589 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:05.940598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:05.940605 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:05.940834 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:05.941204 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:05.941206 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:05.941208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:05.941364 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:05.942012 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:05.942018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:05.942154 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 02:10:05.942319 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 02:10:05.942728 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":100,"thread_start_us":139,"threads_started":1}
I20260806 02:10:05.946255 10529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:05.947028 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.946778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:54990)
0806 02:10:05.946894 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:05.946970 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:10:05.947203 10477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
I20260806 02:10:05.947983 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.947784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:54994)
0806 02:10:05.947883 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:05.947929 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:10:05.948139 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33903: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
I20260806 02:10:05.958042 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:33903
--webserver_interface=127.28.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:05.997432 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } attempt: 1
I20260806 02:10:05.998323 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.998023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55002)
0806 02:10:05.998175 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:05.998242 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:10:05.999146 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:05.998943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55006)
0806 02:10:05.999042 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:05.999088 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:10:05.999311 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33903: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
I20260806 02:10:06.067493 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } attempt: 2
I20260806 02:10:06.068404 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.068101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55020)
0806 02:10:06.068258 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.068324 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:10:06.069253 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.069030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55032)
0806 02:10:06.069137 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.069189 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:10:06.069470 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33903: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33903: connect: Connection refused (error 111)
W20260806 02:10:06.070571 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:06.070825 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:06.070866 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:06.074882 10537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:06.074931 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:06.074951 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:06.074968 10537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:06.074983 10537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:06.080556 10537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:06.081545 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:06.082559 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:06.090814 10544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:06.090814 10546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:06.090909 10543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:06.091126 10537 server_base.cc:1034] running on GCE node
I20260806 02:10:06.091696 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:06.092767 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:06.093963 10537 hybrid_clock.cc:648] HybridClock initialized: now 1785982206093954 us; error 30 us; skew 500 ppm
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:06.096815 10537 init.cc:377] Logged in from keytab as kudu/127.28.21.61@KRBTEST.COM (short username kudu)
I20260806 02:10:06.097697 10537 webserver.cc:460] Webserver started at http://127.28.21.61:46121/ using document root <none> and password file <none>
I20260806 02:10:06.098196 10537 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:06.098253 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:06.098557 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:06.100668 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data/instance:
uuid: "797da497fe4441ed81a8598aabaf9f03"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "e2a76d7336a5b4992f71e95da1df2ea0"
server_key_iv: "2e26a92e78a810ca1a8c4dda92b973de"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.101227 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal/instance:
uuid: "797da497fe4441ed81a8598aabaf9f03"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "e2a76d7336a5b4992f71e95da1df2ea0"
server_key_iv: "2e26a92e78a810ca1a8c4dda92b973de"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.105686 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 02:10:06.108444 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:06.109877 10537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:10:06.110028 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "797da497fe4441ed81a8598aabaf9f03"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "e2a76d7336a5b4992f71e95da1df2ea0"
server_key_iv: "2e26a92e78a810ca1a8c4dda92b973de"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.110142 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:06.125767 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:06.128975 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:06.129184 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:06.136535 10537 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.61:33903
I20260806 02:10:06.136556 10609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.61:33903 every 8 connection(s)
I20260806 02:10:06.137478 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
I20260806 02:10:06.140388 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:06.144511 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10537
I20260806 02:10:06.144634 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-1/wal/instance
I20260806 02:10:06.144428 10610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:06.144971 28756 external_mini_cluster.cc:1428] Setting key c88d47591c8f9eb3055bc3778bf5048a
WARNING: no policy specified for kudu/127.28.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.60@KRBTEST.COM" created.
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:06.160682 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } attempt: 3
I20260806 02:10:06.162860 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.146573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.28.21.62:41353)
0806 02:10:06.146747 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.146752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.147770 (+ 1018us) server_negotiation.cc:407] Connection header received
0806 02:10:06.148697 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.148700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.148764 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.148870 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.150100 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.150709 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.151818 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.152072 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.152180 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.157428 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.157433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.157441 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.157468 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.160016 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.160718 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.160722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.160726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.160851 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.161299 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.161305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.161310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.161610 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.161725 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.162544 (+ 819us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.162690 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:10:06.163735 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.147080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41353 (local address 127.0.0.1:49090)
0806 02:10:06.147620 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.147662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.148493 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.149041 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.149048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.149392 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.149951 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.149972 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.150841 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.150844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.151690 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.151700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.152230 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.152234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.152393 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.152986 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.153001 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.157253 (+ 4252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.160146 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.160155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.160173 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.160571 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.160964 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.160970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.160975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.161174 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.161740 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.161747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.162090 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.162320 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.162710 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":173,"threads_started":1}
I20260806 02:10:06.165540 10610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:10:06.173935 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.161167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55036)
0806 02:10:06.161389 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.161408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.161496 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.162229 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.162235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.162467 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.162635 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.162647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.164218 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.164224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.165136 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.165148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.165994 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.166002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.166128 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.166796 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.166815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.170064 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.172219 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.172222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.172225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.172510 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.172856 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.172859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.172862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.172955 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.173464 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.173467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.173550 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.173640 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.173789 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260806 02:10:06.174151 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.161289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.28.21.61:33903)
0806 02:10:06.161771 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.161785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.161811 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:10:06.161851 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.161856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.162069 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.162225 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.162794 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.164082 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.165271 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.165868 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.166000 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.170208 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.170217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.170227 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.170248 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.172029 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.172651 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.172656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.172657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.172746 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.173047 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.173052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.173054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.173345 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.173490 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.173751 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.173944 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":134,"threads_started":1}
I20260806 02:10:06.175693 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.166095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55038)
0806 02:10:06.166215 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.166230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.166310 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.167074 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.167080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.167230 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.167324 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.167328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.168254 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.168257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.168820 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.168829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.169383 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.169386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.169585 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.170062 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.170073 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.171454 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.174189 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.174195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.174197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.174514 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.174768 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.174773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.174775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.174857 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.175245 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.175247 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.175361 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.175493 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.175592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
I20260806 02:10:06.176240 10529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:06.176919 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:46869
--webserver_interface=127.28.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:06.176970 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.176686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49020)
0806 02:10:06.176820 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.176880 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:10:06.177752 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.177438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49024)
0806 02:10:06.177614 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.177675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260806 02:10:06.177927 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.175860 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.166328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.28.21.61:33903)
0806 02:10:06.166805 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.166817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.166833 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:10:06.166872 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.166875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.166925 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.167032 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.167480 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.168153 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.168946 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.169240 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.169507 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.171634 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.171637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.171639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.171660 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.174071 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.174616 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.174618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.174619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.174681 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.174950 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.174956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.174957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.175162 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.175257 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.175567 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.175674 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":340,"thread_start_us":170,"threads_started":1}
I20260806 02:10:06.179265 10610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:06.179909 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.179681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49028)
0806 02:10:06.179795 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.179859 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:10:06.180136 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.180971 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.180699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49044)
0806 02:10:06.180835 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.180896 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:10:06.181121 10610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.209121 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 1
I20260806 02:10:06.210041 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.209700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49056)
0806 02:10:06.209881 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.209945 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 02:10:06.210948 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.210692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49058)
0806 02:10:06.210826 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.210883 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 02:10:06.211128 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.230372 10610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 1
I20260806 02:10:06.231293 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.230957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49066)
0806 02:10:06.231119 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.231191 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:10:06.232090 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.231844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49072)
0806 02:10:06.231982 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.232027 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 02:10:06.232335 10610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.286288 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 2
I20260806 02:10:06.287250 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.286907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49076)
0806 02:10:06.287087 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.287159 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:10:06.288048 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.287810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49082)
0806 02:10:06.287920 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.287992 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:10:06.288182 10529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
W20260806 02:10:06.294462 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:06.294727 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:06.294828 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:06.298807 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:06.298877 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:06.298934 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:06.298997 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:06.299034 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:06.300479 10610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 2
I20260806 02:10:06.301234 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.300925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49090)
0806 02:10:06.301093 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.301154 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:10:06.301944 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.301723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49100)
0806 02:10:06.301834 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.301885 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:10:06.302102 10610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:46869: Network error: Client connection negotiation failed: client connection to 127.28.21.60:46869: connect: Connection refused (error 111)
I20260806 02:10:06.304807 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:41353,127.28.21.61:33903,127.28.21.60:46869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:46869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:06.305770 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:06.306802 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:06.314961 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:06.314967 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:06.315102 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:06.315222 10617 server_base.cc:1034] running on GCE node
I20260806 02:10:06.315783 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:06.316856 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:06.318055 10617 hybrid_clock.cc:648] HybridClock initialized: now 1785982206318035 us; error 32 us; skew 500 ppm
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:06.320983 10617 init.cc:377] Logged in from keytab as kudu/127.28.21.60@KRBTEST.COM (short username kudu)
I20260806 02:10:06.321928 10617 webserver.cc:460] Webserver started at http://127.28.21.60:44951/ using document root <none> and password file <none>
I20260806 02:10:06.322561 10617 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:06.322660 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:06.322899 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:06.324928 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data/instance:
uuid: "aec2d406565447a697b549ea653cb272"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "32194ee5c31005ed69f7ce65b286a527"
server_key_iv: "1c4ae73595715ec620e9a40ae309231a"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.325497 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal/instance:
uuid: "aec2d406565447a697b549ea653cb272"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "32194ee5c31005ed69f7ce65b286a527"
server_key_iv: "1c4ae73595715ec620e9a40ae309231a"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.329202 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:10:06.331723 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:06.332978 10617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:10:06.333123 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "aec2d406565447a697b549ea653cb272"
format_stamp: "Formatted at 2026-08-06 02:10:06 on dist-test-slave-gp6n"
server_key: "32194ee5c31005ed69f7ce65b286a527"
server_key_iv: "1c4ae73595715ec620e9a40ae309231a"
server_key_version: "encryptionkey@0"
I20260806 02:10:06.333235 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:06.356673 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:06.360070 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:06.360286 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:06.367609 10617 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.60:46869
I20260806 02:10:06.367635 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.60:46869 every 8 connection(s)
I20260806 02:10:06.368557 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
I20260806 02:10:06.371320 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:06.374581 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10617
I20260806 02:10:06.374739 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785982043785074-28756-0/minicluster-data/master-2/wal/instance
I20260806 02:10:06.375072 28756 external_mini_cluster.cc:1428] Setting key 183364cfe93a2fc743dde44f98ac8f0d
I20260806 02:10:06.375344 10689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:10:06.387351 10529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 3
I20260806 02:10:06.388417 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.377387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.28.21.62:41353)
0806 02:10:06.377535 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.377540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.377869 (+ 329us) server_negotiation.cc:407] Connection header received
0806 02:10:06.378217 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.378222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.378278 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.378375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.379339 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.379849 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.380940 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.381303 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.381381 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.384323 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.384326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.384328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.384351 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.385937 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.386544 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.386547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.386548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.386631 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.387017 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.387020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.387023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.387343 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.387455 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.388114 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.388244 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:10:06.389918 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.377314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:41353 (local address 127.0.0.1:49100)
0806 02:10:06.377725 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.377775 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.378011 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.378470 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.378475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.378908 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.379184 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.379203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.379957 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.379960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.380824 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.380833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.381430 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.381434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.381769 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.382484 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.382502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.384164 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.386049 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.386058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.386070 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.386420 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.386729 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.386733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.386735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.386897 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.387460 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.387466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.387800 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.388805 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.389175 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":168,"thread_start_us":130,"threads_started":1}
I20260806 02:10:06.389935 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.376478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.28.21.60:46869)
0806 02:10:06.376905 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.376918 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.376950 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:10:06.377723 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.377746 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.378045 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.378265 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.379214 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.380183 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.381236 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.381873 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.382005 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.384733 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.384742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.384756 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.384783 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.386671 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.387165 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.387169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.387170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.387236 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.387557 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.387560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.387562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.387899 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.388681 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.388803 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.389096 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260806 02:10:06.391526 10689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:06.393282 10610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } attempt: 3
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:10:06.400142 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.387847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49120)
0806 02:10:06.388039 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.388054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.388122 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.388637 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.388640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.388833 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.388926 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.388933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.389959 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.389965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.390749 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.390759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.391839 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.391843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.391943 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.392448 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.392465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.394044 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.397746 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.397749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.397752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.398288 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.398686 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.398691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.398693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.398787 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.399591 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.399594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.399708 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.399827 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.399975 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 02:10:06.400512 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.387919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.28.21.60:46869)
0806 02:10:06.388284 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.388292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.388305 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:10:06.388338 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.388341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.388378 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.388482 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.389044 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.389831 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.390879 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.391696 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.391804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.394279 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.394282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.394285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.394307 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.397615 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.398441 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.398444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.398446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.398565 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.398914 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.398918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.398921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.399134 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.399293 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.399881 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.399999 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":150,"threads_started":1}
I20260806 02:10:06.404325 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.392521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.28.21.62:41353)
0806 02:10:06.392636 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.392641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.393213 (+ 572us) server_negotiation.cc:407] Connection header received
0806 02:10:06.393384 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.393388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.393431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.393499 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.394344 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.395071 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.396378 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.396701 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.396785 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.400176 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.400180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.400183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.400205 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.402112 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.402616 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.402619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.402621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.402723 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.403049 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.403052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.403054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.403264 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.403470 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.404095 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.404195 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:10:06.404326 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.393648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49136)
0806 02:10:06.393912 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.393927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.394012 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.394733 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.394736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.394911 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.394999 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.395004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.396103 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.396106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.396919 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.396928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.397578 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.397582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.397681 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.398119 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.398139 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.400037 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.402613 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.402616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.402618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.402881 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.403235 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.403238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.403240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.403381 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.403792 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.403795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.403892 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.403977 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.404139 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":27}
I20260806 02:10:06.404592 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.392085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33903 (local address 127.0.0.1:55050)
0806 02:10:06.392758 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.392774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.392852 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.393105 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.393108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.393285 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.393403 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.393412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.394488 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.394492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.395443 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.395457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.396338 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.396344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.396458 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.396914 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.396929 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.398586 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.401195 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.401201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.401206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.402630 (+ 1424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.402992 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.402994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.402996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.403099 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.403657 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.403660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.403743 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.404298 (+ 555us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.404431 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":22}
I20260806 02:10:06.404610 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.392598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.28.21.61:33903)
0806 02:10:06.392745 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.392749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.392830 (+ 81us) server_negotiation.cc:407] Connection header received
0806 02:10:06.392937 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.392942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.393001 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.393089 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.393562 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.394322 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.395591 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.396037 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.396170 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.398729 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.398736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.398740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.398765 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.401094 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.402752 (+ 1658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.402757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.402759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.402856 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.403215 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.403222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.403226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.403512 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.403596 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.403919 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.404067 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":64}
I20260806 02:10:06.405071 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.394199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.28.21.60:46869)
0806 02:10:06.394523 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.394527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.394539 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:10:06.394571 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.394574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.394608 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.395103 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.395198 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.395979 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.397054 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.397438 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.397907 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.400176 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.400180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.400182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.400205 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.402488 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.403006 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.403009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.403012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.403123 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.403483 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.403489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.403492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.403679 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.404751 (+ 1072us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.404824 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.404927 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":254}
I20260806 02:10:06.405534 10689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:10:06.404383 10529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: Bootstrap starting.
I20260806 02:10:06.408145 10610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: Bootstrap starting.
I20260806 02:10:06.408488 10529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:06.409401 10529 log.cc:824] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:06.410495 10610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:06.411458 10610 log.cc:824] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:06.411595 10529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: No bootstrap required, opened a new log
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:10:06.414062 10610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: No bootstrap required, opened a new log
I20260806 02:10:06.416028 10529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.416339 10529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:06.416440 10529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a14863bc8243b49329e208df5b72f8, State: Initialized, Role: FOLLOWER
I20260806 02:10:06.416525 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.406333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.28.21.61:33903)
0806 02:10:06.406454 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.406458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.406628 (+ 170us) server_negotiation.cc:407] Connection header received
0806 02:10:06.406721 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.406723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.406763 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.406848 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.407594 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.408277 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.409338 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.409641 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.409739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.412704 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.412708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.412711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.412734 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.414532 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.414991 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.414994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.414996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.415088 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.415421 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.415424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.415426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.415645 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.415832 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.416214 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.416354 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 02:10:06 dist-test-slave-gp6n krb5kdc[10445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:10:06.417016 10529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.418569 10529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:06.419179 10610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.419482 10610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:06.419597 10610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797da497fe4441ed81a8598aabaf9f03, State: Initialized, Role: FOLLOWER
I20260806 02:10:06.420157 10610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.420457 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.410668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:46869 (local address 127.0.0.1:49150)
0806 02:10:06.410976 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:06.410989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:06.411066 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:06.412175 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:06.412180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:06.412376 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:06.412481 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.412488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.413602 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.413608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.414471 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:06.414483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.415132 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.415136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:06.415218 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.415649 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:06.415661 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:06.417071 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:06.418970 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.418973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.418975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.419209 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.419488 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:06.419491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:06.419492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:06.419571 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:06.419992 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:06.419996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:06.420085 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:10:06.420167 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:10:06.420288 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260806 02:10:06.420557 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:10:06.420917 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.410867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.28.21.60:46869)
0806 02:10:06.411966 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.411970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.411984 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:10:06.412018 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.412021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.412061 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.412590 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.412713 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.413513 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.414651 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.415020 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.415416 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.417207 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.417214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.417220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.417249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.418867 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.419310 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.419313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.419315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.419397 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.419668 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.419674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.419679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.419887 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.420623 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.420690 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.420784 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018}
I20260806 02:10:06.421161 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.421311 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.421598 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.421779 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.422626 10610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:10:06.425648 10709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:10:06.425731 10709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:10:06.425885 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: Bootstrap starting.
I20260806 02:10:06.424619 10458 master_runner.cc:186] Error getting master registration for 127.28.21.61:33903: OK, instance_id { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" instance_seqno: 1785982206134554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:10:06.428507 10458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:10:06.428978 10689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:06.429934 10689 log.cc:824] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:06.433259 10537 master_runner.cc:186] Error getting master registration for 127.28.21.60:46869: OK, instance_id { permanent_uuid: "aec2d406565447a697b549ea653cb272" instance_seqno: 1785982206365549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:10:06.435853 10537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:10:06.441622 10731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:10:06.441766 10731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:10:06.442425 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.422986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.28.21.62:41353)
0806 02:10:06.423088 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.423092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.428855 (+ 5763us) server_negotiation.cc:407] Connection header received
0806 02:10:06.429224 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.429227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.429276 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.429355 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.430384 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.431182 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.432442 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.432776 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.432858 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.438060 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.438064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.438068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.438098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.440436 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.440945 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.440949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.440952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.441054 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.441364 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.441368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.441370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.441602 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.441715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.442112 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.442247 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:10:06.442929 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.424070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.28.21.61:33903)
0806 02:10:06.424481 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.424486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.433563 (+ 9077us) server_negotiation.cc:407] Connection header received
0806 02:10:06.433774 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.433778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.433827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.433889 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.434921 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.435709 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.436860 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.437080 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.437196 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.438983 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.438990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.438993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.439022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.440707 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.441174 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.441177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.441179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.441255 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.441557 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.441560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.441562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.441783 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.441914 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.442672 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.442788 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260806 02:10:06.444024 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: No bootstrap required, opened a new log
I20260806 02:10:06.448539 10689 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.448992 10689 raft_consensus.cc:374] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:06.449091 10689 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2d406565447a697b549ea653cb272, State: Initialized, Role: FOLLOWER
I20260806 02:10:06.449672 10689 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.451148 10689 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:06.451702 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:06.426814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.28.21.60:46869)
0806 02:10:06.426910 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:06.426914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:06.442842 (+ 15928us) server_negotiation.cc:407] Connection header received
0806 02:10:06.442889 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:06.442892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:06.442939 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:06.443034 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:06.444009 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.444747 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.445828 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:06.446179 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:06.446260 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:06.447973 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:06.447979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:06.447982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:06.448008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:06.449775 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.450288 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.450292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.450294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.450380 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:06.450721 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:06.450724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:06.450726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:06.450937 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:06.451055 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:06.451422 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 02:10:06.451543 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 02:10:06.457185 10745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:10:06.457440 10745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:10:06.457655 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.457808 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.459429 10617 master_runner.cc:263] No leader master found from master aec2d406565447a697b549ea653cb272
I20260806 02:10:06.459501 10617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:10:06.611665 10640 tablet.cc:2369] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:10:06.630743 10561 tablet.cc:2369] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:10:06.665920 10480 tablet.cc:2369] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:06.692847 10698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:06.693022 10698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.693871 10698 leader_election.cc:288] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 797da497fe4441ed81a8598aabaf9f03 (127.28.21.61:33903), aec2d406565447a697b549ea653cb272 (127.28.21.60:46869)
I20260806 02:10:06.694459 10584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a14863bc8243b49329e208df5b72f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797da497fe4441ed81a8598aabaf9f03" is_pre_election: true
I20260806 02:10:06.694588 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a14863bc8243b49329e208df5b72f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2d406565447a697b549ea653cb272" is_pre_election: true
I20260806 02:10:06.694798 10584 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a14863bc8243b49329e208df5b72f8 in term 0.
I20260806 02:10:06.694980 10661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a14863bc8243b49329e208df5b72f8 in term 0.
I20260806 02:10:06.695288 10477 leader_election.cc:302] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a14863bc8243b49329e208df5b72f8, 797da497fe4441ed81a8598aabaf9f03; no voters:
I20260806 02:10:06.695566 10698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:06.695647 10698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:06.695678 10698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:06.696772 10698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.697114 10698 leader_election.cc:288] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [CANDIDATE]: Term 1 election: Requested vote from peers 797da497fe4441ed81a8598aabaf9f03 (127.28.21.61:33903), aec2d406565447a697b549ea653cb272 (127.28.21.60:46869)
I20260806 02:10:06.697530 10584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a14863bc8243b49329e208df5b72f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797da497fe4441ed81a8598aabaf9f03"
I20260806 02:10:06.697530 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34a14863bc8243b49329e208df5b72f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2d406565447a697b549ea653cb272"
I20260806 02:10:06.697670 10661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:06.697670 10584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:06.699100 10661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a14863bc8243b49329e208df5b72f8 in term 1.
I20260806 02:10:06.699141 10584 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a14863bc8243b49329e208df5b72f8 in term 1.
I20260806 02:10:06.699476 10477 leader_election.cc:302] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a14863bc8243b49329e208df5b72f8, aec2d406565447a697b549ea653cb272; no voters:
I20260806 02:10:06.699674 10698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:06.699975 10698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [term 1 LEADER]: Becoming Leader. State: Replica: 34a14863bc8243b49329e208df5b72f8, State: Running, Role: LEADER
I20260806 02:10:06.700320 10698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } }
I20260806 02:10:06.702299 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a14863bc8243b49329e208df5b72f8. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.702455 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:06.702826 10749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:06.703683 10749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:06.708487 10661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [term 1 FOLLOWER]: Refusing update from remote peer 34a14863bc8243b49329e208df5b72f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:06.708511 10584 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [term 1 FOLLOWER]: Refusing update from remote peer 34a14863bc8243b49329e208df5b72f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:06.709129 10747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:06.709296 10698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:06.713693 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a14863bc8243b49329e208df5b72f8. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.713843 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.713747 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a14863bc8243b49329e208df5b72f8. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.713891 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.716638 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.716730 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:06.716595 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.716887 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:06.716997 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.717036 10751 mvcc.cc:205] Tried to move back new op lower bound from 7315383118671212544 to 7315383118647222272. Current Snapshot: MvccSnapshot[applied={T|T < 7315383118671212544}]
I20260806 02:10:06.717092 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.717422 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34a14863bc8243b49329e208df5b72f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a14863bc8243b49329e208df5b72f8" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 41353 } } peers { permanent_uuid: "797da497fe4441ed81a8598aabaf9f03" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33903 } } peers { permanent_uuid: "aec2d406565447a697b549ea653cb272" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 46869 } } }
I20260806 02:10:06.717552 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:10:06.718045 10749 catalog_manager.cc:1347] Generated new cluster ID: cbb0cf7c63154298b677d5c46b740591
I20260806 02:10:06.718134 10749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:06.736608 10749 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:10:06.737686 10749 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:06.749464 10749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34a14863bc8243b49329e208df5b72f8: Generated new TSK 0
I20260806 02:10:06.750504 10749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:10:07.443117 10731 catalog_manager.cc:1259] Loaded cluster ID: cbb0cf7c63154298b677d5c46b740591
I20260806 02:10:07.443204 10731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: loading cluster ID for follower catalog manager: success
I20260806 02:10:07.444887 10731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: acquiring CA information for follower catalog manager: success
I20260806 02:10:07.445999 10731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 797da497fe4441ed81a8598aabaf9f03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:10:07.458645 10745 catalog_manager.cc:1259] Loaded cluster ID: cbb0cf7c63154298b677d5c46b740591
I20260806 02:10:07.458714 10745 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: loading cluster ID for follower catalog manager: success
I20260806 02:10:07.460237 10745 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: acquiring CA information for follower catalog manager: success
I20260806 02:10:07.461417 10745 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aec2d406565447a697b549ea653cb272: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:10:07.810998 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10458
I20260806 02:10:07.818921 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10537
I20260806 02:10:07.825651 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10617
2026-08-06T02:10:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3313 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:10:07 dist-test-slave-gp6n krb5kdc[10765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:10:07 dist-test-slave-gp6n krb5kdc[10765](info): set up 2 sockets
Aug 06 02:10:07 dist-test-slave-gp6n krb5kdc[10765](info): commencing operation
krb5kdc: starting...
W20260806 02:10:08.905465 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:10:08 dist-test-slave-gp6n krb5kdc[10765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:10:09.033563 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46397
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:09.145520 10778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:09.145816 10778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:09.145900 10778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:09.149957 10778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:09.150036 10778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:09.150074 10778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:09.150130 10778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:09.150168 10778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:09.155432 10778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:09.156468 10778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:09.157480 10778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:09.166000 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:09.166015 10783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:09.166038 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:09.166356 10778 server_base.cc:1034] running on GCE node
I20260806 02:10:09.167003 10778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:09.168088 10778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:09.169271 10778 hybrid_clock.cc:648] HybridClock initialized: now 1785982209169245 us; error 46 us; skew 500 ppm
Aug 06 02:10:09 dist-test-slave-gp6n krb5kdc[10765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:09.172254 10778 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:09.173197 10778 webserver.cc:460] Webserver started at http://127.28.21.62:39815/ using document root <none> and password file <none>
I20260806 02:10:09.173723 10778 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:09.173784 10778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:09.173947 10778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:09.176019 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "433d26d22cdf4052b4db22c2c61cdad3"
format_stamp: "Formatted at 2026-08-06 02:10:09 on dist-test-slave-gp6n"
server_key: "5a2fe10b2fdbd30fd23c7c8818c902e3"
server_key_iv: "ab76ac3781bbf7789a3dec092de6eb2c"
server_key_version: "encryptionkey@0"
I20260806 02:10:09.176532 10778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "433d26d22cdf4052b4db22c2c61cdad3"
format_stamp: "Formatted at 2026-08-06 02:10:09 on dist-test-slave-gp6n"
server_key: "5a2fe10b2fdbd30fd23c7c8818c902e3"
server_key_iv: "ab76ac3781bbf7789a3dec092de6eb2c"
server_key_version: "encryptionkey@0"
I20260806 02:10:09.180132 10778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:10:09.182389 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:09.183646 10778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:09.183780 10778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "433d26d22cdf4052b4db22c2c61cdad3"
format_stamp: "Formatted at 2026-08-06 02:10:09 on dist-test-slave-gp6n"
server_key: "5a2fe10b2fdbd30fd23c7c8818c902e3"
server_key_iv: "ab76ac3781bbf7789a3dec092de6eb2c"
server_key_version: "encryptionkey@0"
I20260806 02:10:09.183883 10778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:09.198513 10778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:09.201777 10778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:09.201967 10778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:09.209074 10778 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:46397
I20260806 02:10:09.209095 10848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:46397 every 8 connection(s)
I20260806 02:10:09.210052 10778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:09.210281 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10778
I20260806 02:10:09.210443 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:10:09.210806 28756 external_mini_cluster.cc:1428] Setting key 7005cb2105f1f925f81656a232e328c9
I20260806 02:10:09.213346 10849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:09.219808 10849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3: Bootstrap starting.
Aug 06 02:10:09 dist-test-slave-gp6n krb5kdc[10765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:09.222504 10849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:09.223392 10849 log.cc:824] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:09.225622 10849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3: No bootstrap required, opened a new log
I20260806 02:10:09.225936 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:09.212232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.28.21.62:46397)
0806 02:10:09.212667 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:09.212686 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:09.212729 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:10:09.213602 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:09.213625 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:09.214020 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:09.214376 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:09.215109 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:09.216626 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:09.217479 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:09.218158 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:09.218329 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:09.220904 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:09.220911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:09.220925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:09.220957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:09.222950 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:09.223356 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:09.223362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:09.223368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:09.223560 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:09.223832 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:09.223835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:09.223836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:09.224271 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:09.224468 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:09.224727 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:10:09.225014 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260806 02:10:09.229159 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:10:09.229097 10849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } }
I20260806 02:10:09.229377 10849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:09.229446 10849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433d26d22cdf4052b4db22c2c61cdad3, State: Initialized, Role: FOLLOWER
I20260806 02:10:09.229885 10849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } }
I20260806 02:10:09.230039 10849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:09.230116 10849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:09.230225 10849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:09.231251 10849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } }
I20260806 02:10:09.231583 10849 leader_election.cc:302] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 433d26d22cdf4052b4db22c2c61cdad3; no voters:
I20260806 02:10:09.231894 10849 leader_election.cc:288] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:10:09.232045 10854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:09.232285 10854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [term 1 LEADER]: Becoming Leader. State: Replica: 433d26d22cdf4052b4db22c2c61cdad3, State: Running, Role: LEADER
I20260806 02:10:09.232648 10854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } }
I20260806 02:10:09.233117 10849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:09.234254 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "433d26d22cdf4052b4db22c2c61cdad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } } }
I20260806 02:10:09.234428 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:09.234849 10865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:09.234792 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 433d26d22cdf4052b4db22c2c61cdad3. Latest consensus state: current_term: 1 leader_uuid: "433d26d22cdf4052b4db22c2c61cdad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "433d26d22cdf4052b4db22c2c61cdad3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46397 } } }
I20260806 02:10:09.235055 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:09.237759 10865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:09.244019 10865 catalog_manager.cc:1347] Generated new cluster ID: 6c29a6f33c4c4159ac3a7607cc1c4411
I20260806 02:10:09.244110 10865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:09.256181 10865 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:10:09.257479 10865 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:09.263744 10865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 433d26d22cdf4052b4db22c2c61cdad3: Generated new TSK 0
I20260806 02:10:09.264500 10865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:10:09.285115 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10778
2026-08-06T02:10:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1458 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:10:09 dist-test-slave-gp6n krb5kdc[10878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:10:09 dist-test-slave-gp6n krb5kdc[10878](info): set up 2 sockets
Aug 06 02:10:09 dist-test-slave-gp6n krb5kdc[10878](info): commencing operation
krb5kdc: starting...
W20260806 02:10:10.353191 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:10:10 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:10:10.481799 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:10.602387 10893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:10.602703 10893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:10.602788 10893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:10.606693 10893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:10.606774 10893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:10.606805 10893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:10.606837 10893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:10.606891 10893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:10.611999 10893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:10.613059 10893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:10.614082 10893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:10.622509 10898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:10.622546 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:10.622546 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:10.622756 10893 server_base.cc:1034] running on GCE node
I20260806 02:10:10.623361 10893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:10.624590 10893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:10.625813 10893 hybrid_clock.cc:648] HybridClock initialized: now 1785982210625780 us; error 46 us; skew 500 ppm
Aug 06 02:10:10 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:10.628736 10893 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:10.629637 10893 webserver.cc:460] Webserver started at http://127.28.21.62:37513/ using document root <none> and password file <none>
I20260806 02:10:10.630148 10893 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:10.630206 10893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:10.630378 10893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:10.632432 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "d2a2a831d9c042e99d276f75cca1637b"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "0fa680357c446686e8591aa2e6db447e"
server_key_iv: "928aa2219e916954415f70e13b5a63e2"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.632952 10893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "d2a2a831d9c042e99d276f75cca1637b"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "0fa680357c446686e8591aa2e6db447e"
server_key_iv: "928aa2219e916954415f70e13b5a63e2"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.636744 10893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 02:10:10.639328 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:10.640699 10893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:10:10.640836 10893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "d2a2a831d9c042e99d276f75cca1637b"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "0fa680357c446686e8591aa2e6db447e"
server_key_iv: "928aa2219e916954415f70e13b5a63e2"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.640934 10893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:10.671885 10893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:10.675282 10893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:10.675453 10893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:10.683004 10893 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:38089
I20260806 02:10:10.683296 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:38089 every 8 connection(s)
I20260806 02:10:10.684070 10893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:10.687109 10964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:10.688820 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10893
I20260806 02:10:10.688925 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:10:10.689270 28756 external_mini_cluster.cc:1428] Setting key 258caa1f566e4cacc2733088ccf16e54
I20260806 02:10:10.694336 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap starting.
I20260806 02:10:10.697124 10964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:10.698037 10964 log.cc:824] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:10:10 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:10.700268 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: No bootstrap required, opened a new log
I20260806 02:10:10.705550 10964 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:10.705847 10964 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:10.705901 10964 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Initialized, Role: FOLLOWER
I20260806 02:10:10.706022 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:10.690749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.28.21.62:38089)
0806 02:10:10.691203 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:10.691224 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:10.691270 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:10:10.692198 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:10.692225 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:10.692718 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:10.693130 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:10.693870 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:10.695529 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:10.696558 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:10.697386 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:10.697564 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:10.700354 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:10.700361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:10.700380 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:10.700413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:10.702574 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:10.703017 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:10.703026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:10.703034 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:10.703200 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:10.703562 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:10.703566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:10.703568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:10.704111 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:10.704352 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:10.704625 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 02:10:10.704968 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":112,"threads_started":1}
I20260806 02:10:10.706445 10964 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:10.706631 10964 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:10.706707 10964 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:10.706822 10964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:10.708104 10964 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:10.708487 10964 leader_election.cc:302] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a2a831d9c042e99d276f75cca1637b; no voters:
I20260806 02:10:10.708878 10964 leader_election.cc:288] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:10:10.709012 10969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:10.709257 10969 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 LEADER]: Becoming Leader. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Running, Role: LEADER
I20260806 02:10:10.709589 10969 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:10.710180 10964 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:10.711746 10970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:10.711802 10971 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a2a831d9c042e99d276f75cca1637b. Latest consensus state: current_term: 1 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:10.711910 10970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:10.711910 10971 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:10.712569 10977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:10.715340 10977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:10.721670 10977 catalog_manager.cc:1347] Generated new cluster ID: c48c7b1bebc642dbb594a7af0731840f
I20260806 02:10:10.721747 10977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:10.749676 10977 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:10:10.750794 10977 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:10.771468 10977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Generated new TSK 0
I20260806 02:10:10.772197 10977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:10:10.807618 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:10.926143 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:10.926429 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:10.926474 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:10.930580 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:10.930644 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:10.930738 10991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:10.936072 10991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:10.937449 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:10.938699 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:10.948737 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:10.948721 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:10.948720 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:10.948910 10991 server_base.cc:1034] running on GCE node
I20260806 02:10:10.949416 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:10.950088 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:10.951277 10991 hybrid_clock.cc:648] HybridClock initialized: now 1785982210951264 us; error 25 us; skew 500 ppm
Aug 06 02:10:10 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:10.958570 10991 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:10.959501 10991 webserver.cc:460] Webserver started at http://127.28.21.1:37275/ using document root <none> and password file <none>
I20260806 02:10:10.960084 10991 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:10.960176 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:10.960407 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:10.962486 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "3a5ba8107453471e895d477ca3ddf755"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "c0b95e59c5eb545d2215e7ae9081ff92"
server_key_iv: "210c71616aa874371644155861660f82"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.963060 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5ba8107453471e895d477ca3ddf755"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "c0b95e59c5eb545d2215e7ae9081ff92"
server_key_iv: "210c71616aa874371644155861660f82"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.966741 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:10:10.969208 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:10.970587 10991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:10.970707 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "3a5ba8107453471e895d477ca3ddf755"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "c0b95e59c5eb545d2215e7ae9081ff92"
server_key_iv: "210c71616aa874371644155861660f82"
server_key_version: "encryptionkey@0"
I20260806 02:10:10.970829 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:10.985548 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:10.988807 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:10.989008 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:10.989593 10991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:10.990391 10991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:10.990515 10991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:10.990585 10991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:10.990633 10991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:10.999518 10991 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:46563
I20260806 02:10:10.999564 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:46563 every 8 connection(s)
I20260806 02:10:11.000392 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:11.005654 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 10991
I20260806 02:10:11.005812 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:10:11.006124 28756 external_mini_cluster.cc:1428] Setting key ea937473efc17e77083fcd84baabd5b8
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:11.014302 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.002135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:43763 (local address 127.28.21.62:38089)
0806 02:10:11.002270 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.002274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.003130 (+ 856us) server_negotiation.cc:407] Connection header received
0806 02:10:11.004490 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.004493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.004551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.004630 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:11.005774 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.006513 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.007854 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.008102 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.008215 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.010701 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:11.010705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:11.010708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:11.010733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:11.012262 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.012755 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.012759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.012761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.012860 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.013190 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.013195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.013196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.013379 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:11.013503 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.013985 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.014122 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:10:11.015138 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.002532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:43763)
0806 02:10:11.002981 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.003025 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.004274 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.004794 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.004802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.005149 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:11.005613 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.005636 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.006637 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.006645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.007731 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.007740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.008252 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.008256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.008420 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.009020 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:11.009037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:11.010548 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:11.012378 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.012384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.012399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.012655 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.012949 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.012953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.012955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.013095 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.013505 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:11.013512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:11.013756 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.013950 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.014267 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:10:11.016396 11127 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:11.016677 11127 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:11.017256 11127 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:10:11.019106 10927 ts_manager.cc:194] Registered new tserver with Master: 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563)
I20260806 02:10:11.020314 10927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:43763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:10:11.037134 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:11.148569 11132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:11.148861 11132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:11.148962 11132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:11.153508 11132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:11.153592 11132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:11.153684 11132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:11.158941 11132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:11.159974 11132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:11.160975 11132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:11.170679 11138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:11.170734 11140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:11.170768 11132 server_base.cc:1034] running on GCE node
W20260806 02:10:11.170691 11137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:11.171380 11132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:11.172020 11132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:11.173203 11132 hybrid_clock.cc:648] HybridClock initialized: now 1785982211173186 us; error 29 us; skew 500 ppm
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:11.176419 11132 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:11.177323 11132 webserver.cc:460] Webserver started at http://127.28.21.2:40221/ using document root <none> and password file <none>
I20260806 02:10:11.178001 11132 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:11.178097 11132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:11.178336 11132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:11.180492 11132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "e8baffcab7154bb19f6f4f87c57f7783"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "519db1558d6f15be1b6658809a4b432c"
server_key_iv: "f38c09933b3f5694a13910cf24e7af2b"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.181090 11132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "e8baffcab7154bb19f6f4f87c57f7783"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "519db1558d6f15be1b6658809a4b432c"
server_key_iv: "f38c09933b3f5694a13910cf24e7af2b"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.185103 11132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:10:11.187600 11147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.188853 11132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:10:11.188989 11132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "e8baffcab7154bb19f6f4f87c57f7783"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "519db1558d6f15be1b6658809a4b432c"
server_key_iv: "f38c09933b3f5694a13910cf24e7af2b"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.189105 11132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:11.205219 11132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:11.208679 11132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:11.208894 11132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:11.209515 11132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:11.210470 11132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:11.210544 11132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.210615 11132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:11.210657 11132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.220261 11132 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:40977
I20260806 02:10:11.220285 11270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:40977 every 8 connection(s)
I20260806 02:10:11.221274 11132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:11.224140 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11132
I20260806 02:10:11.224277 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:10:11.224556 28756 external_mini_cluster.cc:1428] Setting key 7bb79b7fa7453f94314c72aab0616906
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:10:11.236663 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.223247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:43905 (local address 127.28.21.62:38089)
0806 02:10:11.223401 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.223406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.224385 (+ 979us) server_negotiation.cc:407] Connection header received
0806 02:10:11.226017 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.226022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.226083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.226167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:11.227438 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.227996 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.229158 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.229434 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.229546 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.232690 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:11.232696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:11.232702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:11.232739 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:11.234485 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.234996 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.235002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.235007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.235096 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.235459 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.235464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.235467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.235739 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:11.235870 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.236407 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.236536 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:10:11.237561 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.223731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:43905)
0806 02:10:11.224232 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.224286 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.225650 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.226341 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.226357 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.226786 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:11.227282 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.227296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.228127 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.228130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.229032 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.229041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.229586 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.229590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.229841 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.230568 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:11.230593 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:11.232507 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:11.234603 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.234608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.234621 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.234890 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.235188 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.235190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.235192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.235322 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.235860 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:11.235864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:11.236120 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.236322 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.236632 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:10:11.238910 11271 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:11.239132 11271 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:11.239581 11271 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
I20260806 02:10:11.240688 10927 ts_manager.cc:194] Registered new tserver with Master: e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
I20260806 02:10:11.241530 10927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:43905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:10:11.257332 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:11.373118 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:11.373391 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:11.373435 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:11.377731 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:11.377803 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:11.377867 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:11.383266 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:11.384310 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:11.385299 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:11.395375 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:11.395463 11278 server_base.cc:1034] running on GCE node
W20260806 02:10:11.395377 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:11.395377 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:11.396106 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:11.396878 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:11.398089 11278 hybrid_clock.cc:648] HybridClock initialized: now 1785982211398059 us; error 43 us; skew 500 ppm
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:11.400888 11278 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:11.401767 11278 webserver.cc:460] Webserver started at http://127.28.21.3:35615/ using document root <none> and password file <none>
I20260806 02:10:11.402335 11278 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:11.402483 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:11.402724 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:11.404784 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "a893c21fdb824275801a830ce8e27f5d"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "d5156e6e781fbe671efbbbbaaeb78fab"
server_key_iv: "afd0a558fa546f5090aa78b127c82722"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.405342 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "a893c21fdb824275801a830ce8e27f5d"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "d5156e6e781fbe671efbbbbaaeb78fab"
server_key_iv: "afd0a558fa546f5090aa78b127c82722"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.409046 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:10:11.411463 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.412772 11278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:11.412896 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a893c21fdb824275801a830ce8e27f5d"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "d5156e6e781fbe671efbbbbaaeb78fab"
server_key_iv: "afd0a558fa546f5090aa78b127c82722"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.412997 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:11.427888 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:11.430780 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:11.430980 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:11.431550 11278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:11.432358 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:11.432428 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.432528 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:11.432580 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.441232 11278 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:37637
I20260806 02:10:11.441259 11411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:37637 every 8 connection(s)
I20260806 02:10:11.442124 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:11.445047 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11278
I20260806 02:10:11.445161 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:10:11.445504 28756 external_mini_cluster.cc:1428] Setting key ff3f44445235944d34d19190849da581
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:11.455575 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.443868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:35265 (local address 127.28.21.62:38089)
0806 02:10:11.444022 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.444025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.444880 (+ 855us) server_negotiation.cc:407] Connection header received
0806 02:10:11.446682 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.446684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.446730 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.446789 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:11.447983 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.448527 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.449633 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.449843 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.449919 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.452267 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:11.452270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:11.452273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:11.452296 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:11.453755 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.454190 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.454193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.454194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.454267 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.454646 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.454648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.454649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.454819 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:11.454897 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.455337 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.455432 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:10:11.456344 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.444183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:35265)
0806 02:10:11.444680 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.444772 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.446454 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.446983 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.446990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.447332 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:11.447836 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.447854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.448633 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.448635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.449530 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.449538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.449966 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.449969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.450105 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.450705 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:11.450720 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:11.452123 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:11.453853 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.453857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.453868 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.454106 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.454345 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:11.454347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:11.454348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:11.454557 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:11.454910 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:11.454914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:11.455132 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.455275 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.455541 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260806 02:10:11.457430 11412 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:11.457693 11412 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:11.458250 11412 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
I20260806 02:10:11.459416 10926 ts_manager.cc:194] Registered new tserver with Master: a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637)
I20260806 02:10:11.460055 10926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:35265
I20260806 02:10:11.471505 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:11.481365 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.473558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.28.21.62:38089)
0806 02:10:11.473712 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.473717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.473902 (+ 185us) server_negotiation.cc:407] Connection header received
0806 02:10:11.474075 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.474079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.474130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.474213 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:11.475075 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.475672 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.476653 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.476880 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.476995 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.478367 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:11.478375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:11.478378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:11.478451 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:11.479664 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.480087 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.480090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.480092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.480176 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:11.480447 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:11.480451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:11.480453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:11.480687 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:11.480833 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.481073 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.481192 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:10:11.484294 10926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:10:11.487339 10926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:10:11.501741 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.495639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:37637 (local address 127.0.0.1:54888)
0806 02:10:11.496451 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.496513 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.496637 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.497836 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.497838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.497853 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:11.497969 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.497973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.499101 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.499104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.500215 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.500225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.501168 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.501171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.501278 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.501324 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.501438 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.501563 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":84,"thread_start_us":90,"threads_started":1}
I20260806 02:10:11.502197 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.495760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.28.21.3:37637)
0806 02:10:11.497235 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.497244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.497262 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:10:11.497322 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.497325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.497515 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.497670 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:11.498083 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.498941 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.500348 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.501029 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.501140 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.501279 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.501960 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.502055 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":194,"threads_started":1}
I20260806 02:10:11.502498 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.496037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.0.0.1:49094)
0806 02:10:11.496931 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.496942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.497027 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.497802 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.497805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.497823 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:11.497947 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.497962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.499355 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.499358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.500709 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.500719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.502076 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.502079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.502164 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.502191 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.502257 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.502326 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":68,"thread_start_us":82,"threads_started":1}
I20260806 02:10:11.502691 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.496432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.28.21.2:40977)
0806 02:10:11.496842 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.496858 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.496909 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:10:11.497096 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.497102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.497317 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.497482 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:11.497803 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.499028 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.500808 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.501575 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.501676 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.501804 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.502126 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.502216 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":146,"threads_started":1}
I20260806 02:10:11.502712 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.496432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.28.21.1:46563)
0806 02:10:11.497210 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.497226 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.497248 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:10:11.497317 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.497322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.497491 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.497627 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:11.498113 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.499221 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.500835 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.501615 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.501740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.502397 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.502470 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.502561 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":69,"threads_started":1}
I20260806 02:10:11.503142 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.495723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40977 (local address 127.0.0.1:34724)
0806 02:10:11.496817 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.496831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.496936 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.497493 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.497500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.497524 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:11.497677 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.497683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.499172 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.499176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.500663 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.500675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.501701 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.501708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.501795 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.501817 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.501882 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.501950 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":92,"thread_start_us":104,"threads_started":1}
I20260806 02:10:11.505199 11058 tablet_service.cc:1467] Processing CreateTablet for tablet 86d070ed702e41a2836fe0879e342a79 (DEFAULT_TABLE table=test-table [id=466f3d99056b47dd84b941b147003ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:11.505333 11342 tablet_service.cc:1467] Processing CreateTablet for tablet 86d070ed702e41a2836fe0879e342a79 (DEFAULT_TABLE table=test-table [id=466f3d99056b47dd84b941b147003ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:11.505678 11200 tablet_service.cc:1467] Processing CreateTablet for tablet 86d070ed702e41a2836fe0879e342a79 (DEFAULT_TABLE table=test-table [id=466f3d99056b47dd84b941b147003ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:11.506310 11058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d070ed702e41a2836fe0879e342a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:11.506378 11342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d070ed702e41a2836fe0879e342a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:11.506589 11200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d070ed702e41a2836fe0879e342a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:11.512944 11429 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Bootstrap starting.
I20260806 02:10:11.513976 11432 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Bootstrap starting.
I20260806 02:10:11.514770 11431 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Bootstrap starting.
I20260806 02:10:11.515987 11429 tablet_bootstrap.cc:654] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:11.516657 11432 tablet_bootstrap.cc:654] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:11.516884 11429 log.cc:824] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:11.517383 11431 tablet_bootstrap.cc:654] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:11.517541 11432 log.cc:824] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:11.518175 11431 log.cc:824] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:11.519045 11429 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: No bootstrap required, opened a new log
I20260806 02:10:11.519254 11429 ts_tablet_manager.cc:1397] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:10:11.519562 11432 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: No bootstrap required, opened a new log
I20260806 02:10:11.519740 11432 ts_tablet_manager.cc:1397] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:11.520147 11431 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: No bootstrap required, opened a new log
I20260806 02:10:11.520308 11431 ts_tablet_manager.cc:1397] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:11.522861 11429 raft_consensus.cc:348] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.523108 11429 raft_consensus.cc:374] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:11.523161 11429 raft_consensus.cc:729] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5ba8107453471e895d477ca3ddf755, State: Initialized, Role: FOLLOWER
I20260806 02:10:11.523494 11432 raft_consensus.cc:348] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.523612 11429 consensus_queue.cc:260] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.523736 11432 raft_consensus.cc:374] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:11.523793 11432 raft_consensus.cc:729] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a893c21fdb824275801a830ce8e27f5d, State: Initialized, Role: FOLLOWER
I20260806 02:10:11.523954 11431 raft_consensus.cc:348] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.524179 11431 raft_consensus.cc:374] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:11.524245 11431 raft_consensus.cc:729] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8baffcab7154bb19f6f4f87c57f7783, State: Initialized, Role: FOLLOWER
I20260806 02:10:11.524248 11127 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
I20260806 02:10:11.524205 11432 consensus_queue.cc:260] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.524690 11431 consensus_queue.cc:260] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.524829 11412 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
I20260806 02:10:11.525223 11432 ts_tablet_manager.cc:1428] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 02:10:11.525593 11431 ts_tablet_manager.cc:1428] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:10:11.526567 11271 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
I20260806 02:10:11.527810 11429 ts_tablet_manager.cc:1428] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260806 02:10:11.604163 11438 raft_consensus.cc:482] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:11.604419 11438 raft_consensus.cc:504] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.605738 11438 leader_election.cc:288] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563)
I20260806 02:10:11.611212 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.606162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:43929 (local address 127.28.21.1:46563)
0806 02:10:11.606296 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.606301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.606564 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:10:11.606656 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.606659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.606710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.606778 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:11.607054 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.607790 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.609784 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.610438 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.610547 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.610615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.610982 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.611088 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:10:11.611253 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.605955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:37637 (local address 127.28.21.2:53053)
0806 02:10:11.606180 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.606195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.606296 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.606632 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.606635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.606653 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:11.606785 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.606794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.607894 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.607897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.609644 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.609655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.610594 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.610599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.610794 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.610854 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.610951 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.611106 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":317}
I20260806 02:10:11.611260 11439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.606058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.28.21.2:43929)
0806 02:10:11.606465 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.606477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:11.606554 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:11.606808 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:11.606812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:11.606832 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:11.606951 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.606956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.607894 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.607897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.609644 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:11.609655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.610594 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.610599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:11.610790 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.610852 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 02:10:11.610951 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:10:11.611106 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":297,"thread_start_us":153,"threads_started":1}
I20260806 02:10:11.611711 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.606089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:53053 (local address 127.28.21.3:37637)
0806 02:10:11.606207 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.606214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.606236 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:10:11.606439 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.606441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.606497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.606604 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:11.606945 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.607774 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.609784 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.610426 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.610548 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.610614 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.611519 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.611595 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:10:11.612315 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d070ed702e41a2836fe0879e342a79" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ba8107453471e895d477ca3ddf755" is_pre_election: true
I20260806 02:10:11.612315 11364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d070ed702e41a2836fe0879e342a79" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a893c21fdb824275801a830ce8e27f5d" is_pre_election: true
I20260806 02:10:11.612650 11078 raft_consensus.cc:2454] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 0.
I20260806 02:10:11.612650 11364 raft_consensus.cc:2454] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 0.
I20260806 02:10:11.613145 11150 leader_election.cc:302] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783; no voters:
I20260806 02:10:11.613446 11438 raft_consensus.cc:2781] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:11.613556 11438 raft_consensus.cc:482] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:11.613618 11438 raft_consensus.cc:3037] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:11.614768 11438 raft_consensus.cc:504] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.615129 11438 leader_election.cc:288] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 election: Requested vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563)
I20260806 02:10:11.615525 11078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d070ed702e41a2836fe0879e342a79" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ba8107453471e895d477ca3ddf755"
I20260806 02:10:11.615525 11364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d070ed702e41a2836fe0879e342a79" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a893c21fdb824275801a830ce8e27f5d"
I20260806 02:10:11.615653 11078 raft_consensus.cc:3037] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:11.615653 11364 raft_consensus.cc:3037] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:11.616945 11078 raft_consensus.cc:2454] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 1.
I20260806 02:10:11.616945 11364 raft_consensus.cc:2454] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 1.
I20260806 02:10:11.617249 11150 leader_election.cc:302] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783; no voters:
I20260806 02:10:11.617416 11438 raft_consensus.cc:2781] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:11.617664 11438 raft_consensus.cc:686] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 LEADER]: Becoming Leader. State: Replica: e8baffcab7154bb19f6f4f87c57f7783, State: Running, Role: LEADER
I20260806 02:10:11.617993 11438 consensus_queue.cc:237] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:11.621312 10925 catalog_manager.cc:5495] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "e8baffcab7154bb19f6f4f87c57f7783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } health_report { overall_health: UNKNOWN } } }
I20260806 02:10:11.655763 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.652302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.28.21.2:40977)
0806 02:10:11.652468 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:11.652471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:11.652485 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:10:11.652562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:11.652564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:11.652619 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:11.652711 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:11.652971 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.653498 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.654343 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:11.654599 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:11.654685 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:11.654963 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:11.655381 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:11.655500 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:11.655574 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:10:11.655626 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:10:11.661692 11364 raft_consensus.cc:1261] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Refusing update from remote peer e8baffcab7154bb19f6f4f87c57f7783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:11.661691 11078 raft_consensus.cc:1261] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Refusing update from remote peer e8baffcab7154bb19f6f4f87c57f7783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:11.662384 11438 consensus_queue.cc:1035] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:11.662643 11440 consensus_queue.cc:1035] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:11.670082 11443 mvcc.cc:205] Tried to move back new op lower bound from 7315383138959122432 to 7315383138790211584. Current Snapshot: MvccSnapshot[applied={T|T < 7315383138959122432}]
I20260806 02:10:11.681103 10925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:11.681298 10925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:11.684043 10925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Sending DeleteTablet for 3 replicas of tablet 86d070ed702e41a2836fe0879e342a79
I20260806 02:10:11.684840 11342 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d070ed702e41a2836fe0879e342a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:11 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54888
I20260806 02:10:11.685351 11058 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d070ed702e41a2836fe0879e342a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:11 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:49094
I20260806 02:10:11.685784 11454 tablet_replica.cc:330] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: stopping tablet replica
I20260806 02:10:11.686082 11454 raft_consensus.cc:2229] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:11.686364 11454 raft_consensus.cc:2258] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:11.687922 11454 ts_tablet_manager.cc:1905] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:11.690716 11454 ts_tablet_manager.cc:1918] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:11.690806 11454 log.cc:1196] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/86d070ed702e41a2836fe0879e342a79
I20260806 02:10:11.690871 11200 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d070ed702e41a2836fe0879e342a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:11 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:34724
I20260806 02:10:11.691103 11454 ts_tablet_manager.cc:1939] T 86d070ed702e41a2836fe0879e342a79 P 3a5ba8107453471e895d477ca3ddf755: Deleting consensus metadata
I20260806 02:10:11.691543 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10991
I20260806 02:10:11.698645 11453 tablet_replica.cc:330] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: stopping tablet replica
I20260806 02:10:11.699002 11453 raft_consensus.cc:2229] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:11.699347 11453 raft_consensus.cc:2258] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:10:11.700073 10913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:46563 (error 108)
I20260806 02:10:11.700920 11453 ts_tablet_manager.cc:1905] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:11.700966 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11132
I20260806 02:10:11.703233 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.702907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.0.0.1:49108)
0806 02:10:11.703049 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.703131 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:46563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 02:10:11.704059 10913 catalog_manager.cc:4584] TS 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86d070ed702e41a2836fe0879e342a79: Network error: Client connection negotiation failed: client connection to 127.28.21.1:46563: connect: Connection refused (error 111)
I20260806 02:10:11.704303 11453 ts_tablet_manager.cc:1918] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:11.704432 11453 log.cc:1196] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/86d070ed702e41a2836fe0879e342a79
I20260806 02:10:11.704782 11453 ts_tablet_manager.cc:1939] T 86d070ed702e41a2836fe0879e342a79 P a893c21fdb824275801a830ce8e27f5d: Deleting consensus metadata
I20260806 02:10:11.705890 10911 catalog_manager.cc:4841] TS a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): tablet 86d070ed702e41a2836fe0879e342a79 (table test-table [id=466f3d99056b47dd84b941b147003ce6]) successfully deleted
I20260806 02:10:11.708976 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11278
I20260806 02:10:11.709293 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:11.708927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40977 (local address 127.0.0.1:34740)
0806 02:10:11.709125 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:11.709191 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.2:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:10:11.715843 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 10893
I20260806 02:10:11.723217 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--webserver_interface=127.28.21.62
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:11.836609 11457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:11.836921 11457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:11.836993 11457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:11.841125 11457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:11.841216 11457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:11.841253 11457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:11.841291 11457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:11.841322 11457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:11.846655 11457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:11.847713 11457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:11.848744 11457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:11.863411 11463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:11.863427 11465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:11.863423 11467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:11.863713 11457 server_base.cc:1034] running on GCE node
I20260806 02:10:11.864661 11457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:11.865792 11457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:11.866953 11457 hybrid_clock.cc:648] HybridClock initialized: now 1785982211866936 us; error 30 us; skew 500 ppm
Aug 06 02:10:11 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:11.869719 11457 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:11.870739 11457 webserver.cc:460] Webserver started at http://127.28.21.62:37513/ using document root <none> and password file <none>
I20260806 02:10:11.871294 11457 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:11.871366 11457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:11.875504 11457 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:10:11.877877 11474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:11.879235 11457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:11.879379 11457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "d2a2a831d9c042e99d276f75cca1637b"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "0fa680357c446686e8591aa2e6db447e"
server_key_iv: "928aa2219e916954415f70e13b5a63e2"
server_key_version: "encryptionkey@0"
I20260806 02:10:11.880008 11457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:11.897699 11457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:11.900591 11457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:11.900804 11457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:11.908102 11457 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:38089
I20260806 02:10:11.908404 11529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:38089 every 8 connection(s)
I20260806 02:10:11.909235 11457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:11.909814 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11457
I20260806 02:10:11.910264 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:46563
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=37275
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:11.914271 11530 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:10:11.916046 11530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap starting.
I20260806 02:10:11.933390 11530 log.cc:824] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:11.941992 11530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:11.942512 11530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap complete.
I20260806 02:10:11.946776 11530 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:11.947199 11530 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Initialized, Role: FOLLOWER
I20260806 02:10:11.947664 11530 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:11.947809 11530 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:11.947899 11530 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:11.948025 11530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:10:11.949878 11530 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:11.950217 11530 leader_election.cc:302] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a2a831d9c042e99d276f75cca1637b; no voters:
I20260806 02:10:11.950595 11530 leader_election.cc:288] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:10:11.950706 11534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:10:11.950901 11534 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 LEADER]: Becoming Leader. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Running, Role: LEADER
I20260806 02:10:11.951186 11534 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:11.951817 11530 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:11.952105 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a2a831d9c042e99d276f75cca1637b. Latest consensus state: current_term: 2 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:11.952225 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:11.952586 11540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:11.952688 11535 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:11.952869 11535 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:11.957288 11540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:11.958122 11540 catalog_manager.cc:1259] Loaded cluster ID: c48c7b1bebc642dbb594a7af0731840f
I20260806 02:10:11.958185 11540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:11.959613 11540 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:10:11.964454 11540 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:10:11.969782 11540 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:11.970546 11540 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Loaded TSK: 0
I20260806 02:10:11.971709 11540 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:10:12.026384 11532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:12.026727 11532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:12.026809 11532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:12.030911 11532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:12.031008 11532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:12.031100 11532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:12.036492 11532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:46563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=37275
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:12.037671 11532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:12.038833 11532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:12.049996 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.049989 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.050454 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:12.050683 11532 server_base.cc:1034] running on GCE node
I20260806 02:10:12.051142 11532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:12.051961 11532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:12.053177 11532 hybrid_clock.cc:648] HybridClock initialized: now 1785982212053149 us; error 36 us; skew 500 ppm
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:12.056926 11532 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:12.058135 11532 webserver.cc:460] Webserver started at http://127.28.21.1:37275/ using document root <none> and password file <none>
I20260806 02:10:12.058907 11532 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:12.059032 11532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:12.064133 11532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:10:12.066772 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.068380 11532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:12.068523 11532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "3a5ba8107453471e895d477ca3ddf755"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "c0b95e59c5eb545d2215e7ae9081ff92"
server_key_iv: "210c71616aa874371644155861660f82"
server_key_version: "encryptionkey@0"
I20260806 02:10:12.069253 11532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:12.080602 11532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:12.083976 11532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:12.084281 11532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:12.084959 11532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:12.085894 11532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:12.085963 11532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.086030 11532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:12.086072 11532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.097471 11532 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:46563
I20260806 02:10:12.097508 11689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:46563 every 8 connection(s)
I20260806 02:10:12.098671 11532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:12.107863 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11532
I20260806 02:10:12.108314 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:40977
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=40221
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:12.117025 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.101014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:58263)
0806 02:10:12.101534 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.101587 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.102873 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.104170 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.104176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.104526 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:12.104989 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.105004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.106807 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.106810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.107681 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.107691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.108676 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.108683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.108933 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.109582 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:12.109608 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:12.111270 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:12.113922 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.113927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.113938 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.114194 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.114655 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.114658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.114659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.114806 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.115596 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:12.115601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:12.115823 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.115980 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.116239 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":171,"threads_started":1}
I20260806 02:10:12.117887 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.100740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:58263 (local address 127.28.21.62:38089)
0806 02:10:12.101176 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.101198 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.101713 (+ 515us) server_negotiation.cc:407] Connection header received
0806 02:10:12.103324 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.103340 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.103734 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.104103 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:12.105176 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.106676 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.107821 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.108357 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.108569 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.111480 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:12.111490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:12.111510 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:12.111549 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:12.113774 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.114319 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.114328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.114337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.114550 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.114908 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.114914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.114918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.115475 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:12.115640 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.116163 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.116524 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":135,"threads_started":1}
I20260806 02:10:12.120127 11690 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:12.120373 11690 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:12.120867 11690 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
I20260806 02:10:12.122372 11491 ts_manager.cc:194] Registered new tserver with Master: 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563)
I20260806 02:10:12.123617 11491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:58263
W20260806 02:10:12.229609 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:12.229892 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:12.229969 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:12.234035 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:12.234114 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:12.234205 11694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:12.239500 11694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:40977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=40221
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:12.240453 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:12.241413 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:12.251202 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.251214 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.251214 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:12.251561 11694 server_base.cc:1034] running on GCE node
I20260806 02:10:12.251959 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:12.252671 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:12.253854 11694 hybrid_clock.cc:648] HybridClock initialized: now 1785982212253839 us; error 27 us; skew 500 ppm
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:12.256657 11694 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:12.257542 11694 webserver.cc:460] Webserver started at http://127.28.21.2:40221/ using document root <none> and password file <none>
I20260806 02:10:12.258100 11694 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:12.258209 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:12.262017 11694 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260806 02:10:12.264298 11709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.265497 11694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:10:12.265630 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "e8baffcab7154bb19f6f4f87c57f7783"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "519db1558d6f15be1b6658809a4b432c"
server_key_iv: "f38c09933b3f5694a13910cf24e7af2b"
server_key_version: "encryptionkey@0"
I20260806 02:10:12.266188 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:12.287282 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:12.290277 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:12.290519 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:12.291159 11694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:12.292250 11716 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:10:12.296705 11694 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:10:12.296784 11694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 02:10:12.296862 11694 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:10:12.298484 11694 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:10:12.298550 11694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 02:10:12.298825 11716 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Bootstrap starting.
I20260806 02:10:12.308604 11694 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:40977
I20260806 02:10:12.308723 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:40977 every 8 connection(s)
I20260806 02:10:12.309584 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:12.316706 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11694
I20260806 02:10:12.317173 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:37637
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=35615
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:12.320076 11716 log.cc:824] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:12.324879 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.311538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:42991 (local address 127.28.21.62:38089)
0806 02:10:12.311676 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.311680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.312403 (+ 723us) server_negotiation.cc:407] Connection header received
0806 02:10:12.313301 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.313305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.313366 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.313460 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:12.314655 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.315189 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.316335 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.316709 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.316829 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.320125 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:12.320128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:12.320130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:12.320158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:12.322219 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.322899 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.322907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.322912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.323030 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.323458 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.323462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.323464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.323701 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:12.323815 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.324574 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.324713 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:10:12.326140 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.311722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:42991)
0806 02:10:12.312251 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.312293 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.313095 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.313632 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.313638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.313986 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:12.314515 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.314531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.315333 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.315338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.316207 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.316216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.316897 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.316902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.317199 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.318014 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:12.318036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:12.319941 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:12.322362 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.322368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.322383 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.322772 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.323158 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.323161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.323162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.323334 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.323840 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:12.323850 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:12.324226 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.324477 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.324907 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":169,"threads_started":1}
I20260806 02:10:12.327551 11828 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:12.327778 11828 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:12.328298 11828 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
I20260806 02:10:12.328562 11716 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:12.328940 11716 tablet_bootstrap.cc:492] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Bootstrap complete.
I20260806 02:10:12.330297 11716 ts_tablet_manager.cc:1397] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.009s
I20260806 02:10:12.330559 11491 ts_manager.cc:194] Registered new tserver with Master: e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
I20260806 02:10:12.333231 11716 raft_consensus.cc:348] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:12.333786 11716 raft_consensus.cc:729] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8baffcab7154bb19f6f4f87c57f7783, State: Initialized, Role: FOLLOWER
I20260806 02:10:12.334286 11716 consensus_queue.cc:260] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } } peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } }
I20260806 02:10:12.334751 11491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:42991
I20260806 02:10:12.335309 11716 ts_tablet_manager.cc:1428] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20260806 02:10:12.336923 11828 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.2@KRBTEST.COM
I20260806 02:10:12.345631 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.333551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40977 (local address 127.0.0.1:34752)
0806 02:10:12.334926 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.334984 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.335122 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.335779 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.335785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.336210 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:12.336336 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.336344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.337862 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.337866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.338794 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.338804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.339631 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.339634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.339712 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.340300 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:12.340312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:12.341620 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:12.343894 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.343904 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.343908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.344141 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.344509 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.344511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.344513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.344597 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.345128 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:12.345133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:12.345230 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.345341 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.345502 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":88,"threads_started":1}
I20260806 02:10:12.345805 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.333673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.28.21.2:40977)
0806 02:10:12.334616 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.334629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.335254 (+ 625us) server_negotiation.cc:407] Connection header received
0806 02:10:12.335326 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.335331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.335608 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.335747 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:12.336469 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.337737 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.338921 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.339510 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.339619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.341765 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:12.341775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:12.341789 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:12.341818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:12.343776 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.344261 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.344269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.344271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.344393 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.344697 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.344702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.344707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.345006 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:12.345146 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.345465 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.345624 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":101,"threads_started":1}
I20260806 02:10:12.346794 11759 tablet_service.cc:1514] Processing DeleteTablet for tablet 86d070ed702e41a2836fe0879e342a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:11 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:34752
I20260806 02:10:12.347652 11837 tablet_replica.cc:330] stopping tablet replica
I20260806 02:10:12.347983 11837 raft_consensus.cc:2229] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:12.348244 11837 raft_consensus.cc:2258] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:12.349344 11837 ts_tablet_manager.cc:1905] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:12.354152 11837 ts_tablet_manager.cc:1918] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:12.354266 11837 log.cc:1196] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/86d070ed702e41a2836fe0879e342a79
I20260806 02:10:12.354640 11837 ts_tablet_manager.cc:1939] T 86d070ed702e41a2836fe0879e342a79 P e8baffcab7154bb19f6f4f87c57f7783: Deleting consensus metadata
I20260806 02:10:12.356676 11477 catalog_manager.cc:4841] TS e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): tablet 86d070ed702e41a2836fe0879e342a79 (table test-table [id=466f3d99056b47dd84b941b147003ce6]) successfully deleted
W20260806 02:10:12.444169 11831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:12.444443 11831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:12.444487 11831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:12.448642 11831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:12.448698 11831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:12.448755 11831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:12.454151 11831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=35615
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:12.455174 11831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:12.456131 11831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:12.465864 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:12.465989 11831 server_base.cc:1034] running on GCE node
W20260806 02:10:12.465886 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.465934 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:12.466652 11831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:12.467353 11831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:12.468621 11831 hybrid_clock.cc:648] HybridClock initialized: now 1785982212468604 us; error 38 us; skew 500 ppm
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:12.471946 11831 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:12.473078 11831 webserver.cc:460] Webserver started at http://127.28.21.3:35615/ using document root <none> and password file <none>
I20260806 02:10:12.473738 11831 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:12.473853 11831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:12.477993 11831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:12.480257 11853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.481551 11831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:12.481698 11831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a893c21fdb824275801a830ce8e27f5d"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "d5156e6e781fbe671efbbbbaaeb78fab"
server_key_iv: "afd0a558fa546f5090aa78b127c82722"
server_key_version: "encryptionkey@0"
I20260806 02:10:12.482259 11831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:12.509793 11831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:12.512707 11831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:12.512916 11831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:12.513516 11831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:12.514325 11831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:12.514395 11831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.514518 11831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:12.514573 11831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.523402 11831 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:37637
I20260806 02:10:12.523430 11970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:37637 every 8 connection(s)
I20260806 02:10:12.524402 11831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:12.527412 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 11831
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:12.536240 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.528130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.28.21.62:38089)
0806 02:10:12.528371 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.528375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.528551 (+ 176us) server_negotiation.cc:407] Connection header received
0806 02:10:12.528717 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.528720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.528766 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.528864 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:12.529656 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.530195 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.531126 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.531355 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.531470 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.533014 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:12.533018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:12.533020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:12.533051 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:12.534581 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.535021 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.535025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.535027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.535111 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.535388 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.535392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.535393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.535579 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:12.535803 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.535942 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.536060 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260806 02:10:12.538146 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.526155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:49869 (local address 127.28.21.62:38089)
0806 02:10:12.526292 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.526296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.527185 (+ 889us) server_negotiation.cc:407] Connection header received
0806 02:10:12.528835 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.528839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.528887 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.528963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:12.530112 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.530693 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.531796 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.532017 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.532125 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.534439 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:12.534442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:12.534445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:12.534492 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:12.535855 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.536489 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.536493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.536494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.536580 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:12.536996 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:12.536999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:12.537000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:12.537223 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:12.537316 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.537874 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.537989 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:10:12.539069 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.526594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:49869)
0806 02:10:12.527034 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.527078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.528629 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.529139 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.529147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.529499 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:12.529967 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.529980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.530812 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.530816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.531671 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.531680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.532161 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.532165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.532328 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.532860 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:12.532874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:12.534265 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:12.535990 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.535995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.536009 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.536355 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.536681 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:12.536684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:12.536687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:12.536897 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:12.537322 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:12.537327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:12.537634 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.537838 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.538168 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260806 02:10:12.540205 11971 heartbeater.cc:347] Connected to a master server at 127.28.21.62:38089
I20260806 02:10:12.540416 11971 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:12.540841 11971 heartbeater.cc:511] Master 127.28.21.62:38089 requested a full tablet report, sending...
I20260806 02:10:12.541772 11491 ts_manager.cc:194] Registered new tserver with Master: a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637)
I20260806 02:10:12.542318 11491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:49869
I20260806 02:10:12.549703 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:12.554143 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.550303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.28.21.62:38089)
0806 02:10:12.550490 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.550495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.550654 (+ 159us) server_negotiation.cc:407] Connection header received
0806 02:10:12.550835 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.550838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.550940 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.551048 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:12.551292 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.551800 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.552732 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.552957 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.553072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.553400 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:12.553768 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:12.553898 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.553979 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.554023 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:10:12.554858 11491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:10:12.558037 11491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:10:12.565470 11759 tablet_service.cc:1467] Processing CreateTablet for tablet 68a401dc5811466da2e418e0cbaa2511 (DEFAULT_TABLE table=test-table [id=c3e7286e460540659bc5955f9b35b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:12.566150 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a401dc5811466da2e418e0cbaa2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:12.569008 11716 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Bootstrap starting.
I20260806 02:10:12.569914 11716 tablet_bootstrap.cc:654] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:12.570190 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.564939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.28.21.1:46563)
0806 02:10:12.565410 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.565422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.565458 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:10:12.565537 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.565542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.565758 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.565949 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:12.566283 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.567167 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.568660 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.569544 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.569711 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.569864 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.569966 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.570060 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":153,"threads_started":1}
I20260806 02:10:12.570168 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.564371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.0.0.1:49112)
0806 02:10:12.564535 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.564550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.564632 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.565928 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.565931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.565959 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:12.566130 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.566138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.567308 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.567314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.568509 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.568519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.569710 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.569714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.569808 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.569848 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.569931 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.570033 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":33,"spinlock_wait_cycles":1408}
I20260806 02:10:12.570616 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.564742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:37637 (local address 127.0.0.1:54896)
0806 02:10:12.565017 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.565031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.565117 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.565956 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.565959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.565978 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:12.566110 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.566116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.567965 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.567969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.568991 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.569001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.570183 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.570188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.570301 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.570326 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.570375 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.570487 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260806 02:10:12.571494 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.565108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.28.21.3:37637)
0806 02:10:12.565480 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.565493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.565520 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:10:12.565597 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.565602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.565791 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.566285 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:12.566477 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.567805 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.569135 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.570013 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.570803 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.570944 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.571006 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.571232 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
I20260806 02:10:12.572243 11716 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: No bootstrap required, opened a new log
I20260806 02:10:12.572342 11716 ts_tablet_manager.cc:1397] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:10:12.572775 11716 raft_consensus.cc:348] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.572898 11716 raft_consensus.cc:374] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:12.572945 11716 raft_consensus.cc:729] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8baffcab7154bb19f6f4f87c57f7783, State: Initialized, Role: FOLLOWER
I20260806 02:10:12.573079 11716 consensus_queue.cc:260] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.573451 11716 ts_tablet_manager.cc:1428] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:10:12.573935 11620 tablet_service.cc:1467] Processing CreateTablet for tablet 68a401dc5811466da2e418e0cbaa2511 (DEFAULT_TABLE table=test-table [id=c3e7286e460540659bc5955f9b35b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:12.574359 11903 tablet_service.cc:1467] Processing CreateTablet for tablet 68a401dc5811466da2e418e0cbaa2511 (DEFAULT_TABLE table=test-table [id=c3e7286e460540659bc5955f9b35b97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:12.575138 11620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a401dc5811466da2e418e0cbaa2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:12.575400 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a401dc5811466da2e418e0cbaa2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:12.581704 11985 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Bootstrap starting.
I20260806 02:10:12.581740 11986 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Bootstrap starting.
I20260806 02:10:12.584116 11985 tablet_bootstrap.cc:654] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:12.584597 11986 tablet_bootstrap.cc:654] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:12.584697 11836 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:12.584884 11836 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.585391 11985 log.cc:824] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:12.585419 11986 log.cc:824] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:12.586154 11836 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563), a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637)
I20260806 02:10:12.587703 11986 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: No bootstrap required, opened a new log
I20260806 02:10:12.587915 11986 ts_tablet_manager.cc:1397] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:12.588176 11985 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: No bootstrap required, opened a new log
I20260806 02:10:12.588366 11985 ts_tablet_manager.cc:1397] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:10:12.590850 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.586295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.28.21.2:41297)
0806 02:10:12.586454 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.586471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.586568 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.586866 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.586869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.586887 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:12.587031 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.587040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.587988 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.587992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.589196 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.589208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.590339 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.590343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.590491 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.590536 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.590611 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.590696 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:10:12.591217 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.586543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:41297 (local address 127.28.21.1:46563)
0806 02:10:12.586656 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.586660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.586674 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:10:12.586720 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.586723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.586771 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.586856 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:12.587139 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.587849 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.589514 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.590109 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.590212 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.590275 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.591012 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.591095 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:10:12.591730 11641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a401dc5811466da2e418e0cbaa2511" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ba8107453471e895d477ca3ddf755" is_pre_election: true
W20260806 02:10:12.592557 11713 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563): Illegal state: must be running to vote when last-logged opid is not known
I20260806 02:10:12.592684 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.586846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:56807 (local address 127.28.21.3:37637)
0806 02:10:12.586956 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.586961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.587255 (+ 294us) server_negotiation.cc:407] Connection header received
0806 02:10:12.587426 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.587429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.587476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.587523 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:12.588081 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.588757 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.591517 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.592121 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.592232 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.592288 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.592509 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.592581 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:10:12.592792 11985 raft_consensus.cc:348] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.592788 11986 raft_consensus.cc:348] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.593031 11985 raft_consensus.cc:374] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:12.593037 11986 raft_consensus.cc:374] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:12.593088 11986 raft_consensus.cc:729] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a893c21fdb824275801a830ce8e27f5d, State: Initialized, Role: FOLLOWER
I20260806 02:10:12.593087 11985 raft_consensus.cc:729] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5ba8107453471e895d477ca3ddf755, State: Initialized, Role: FOLLOWER
I20260806 02:10:12.593216 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.586471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:37637 (local address 127.28.21.2:56807)
0806 02:10:12.587183 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.587196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:12.587314 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:12.587653 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:12.587656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:12.587673 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:12.587883 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.587892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.588969 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.588973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.591256 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:12.591388 (+ 132us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.592252 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.592256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:12.592357 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.592383 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:10:12.592981 (+ 598us) client_negotiation.cc:240] Negotiation successful
0806 02:10:12.593069 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":119,"threads_started":1}
I20260806 02:10:12.593562 11986 consensus_queue.cc:260] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.593591 11985 consensus_queue.cc:260] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.593914 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a401dc5811466da2e418e0cbaa2511" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a893c21fdb824275801a830ce8e27f5d" is_pre_election: true
I20260806 02:10:12.594169 11924 raft_consensus.cc:2454] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 0.
I20260806 02:10:12.594267 11971 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
I20260806 02:10:12.594620 11986 ts_tablet_manager.cc:1428] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260806 02:10:12.594635 11985 ts_tablet_manager.cc:1428] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:10:12.594605 11711 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783; no voters: 3a5ba8107453471e895d477ca3ddf755
I20260806 02:10:12.594930 11690 heartbeater.cc:503] Master 127.28.21.62:38089 was elected leader, sending a full tablet report...
I20260806 02:10:12.595000 11836 raft_consensus.cc:2781] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:12.595086 11836 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:12.595130 11836 raft_consensus.cc:3037] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:12.596521 11836 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:12.597163 11641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a401dc5811466da2e418e0cbaa2511" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ba8107453471e895d477ca3ddf755"
I20260806 02:10:12.597309 11641 raft_consensus.cc:3037] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:12.598716 11641 raft_consensus.cc:2454] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 1.
I20260806 02:10:12.598989 11836 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 election: Requested vote from peers 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563), a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637)
I20260806 02:10:12.599013 11713 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755, e8baffcab7154bb19f6f4f87c57f7783; no voters:
I20260806 02:10:12.599187 11836 raft_consensus.cc:2781] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:12.599314 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a401dc5811466da2e418e0cbaa2511" candidate_uuid: "e8baffcab7154bb19f6f4f87c57f7783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a893c21fdb824275801a830ce8e27f5d"
I20260806 02:10:12.599424 11836 raft_consensus.cc:686] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 LEADER]: Becoming Leader. State: Replica: e8baffcab7154bb19f6f4f87c57f7783, State: Running, Role: LEADER
I20260806 02:10:12.599442 11924 raft_consensus.cc:3037] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:12.599670 11836 consensus_queue.cc:237] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
W20260806 02:10:12.599838 11691 tablet.cc:2369] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:12.600641 11924 raft_consensus.cc:2454] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8baffcab7154bb19f6f4f87c57f7783 in term 1.
I20260806 02:10:12.602367 11491 catalog_manager.cc:5495] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "e8baffcab7154bb19f6f4f87c57f7783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } health_report { overall_health: HEALTHY } } }
I20260806 02:10:12.612675 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.609214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.28.21.2:40977)
0806 02:10:12.609353 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:12.609359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:12.609375 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:10:12.609455 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:12.609458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:12.609506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:12.609615 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:12.609861 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.610371 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.611279 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:12.611527 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:12.611641 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:12.611913 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:12.612287 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:12.612413 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:12.612483 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:10:12.612536 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:10:12.616286 11924 raft_consensus.cc:1261] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Refusing update from remote peer e8baffcab7154bb19f6f4f87c57f7783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:12.616282 11641 raft_consensus.cc:1261] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Refusing update from remote peer e8baffcab7154bb19f6f4f87c57f7783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:12.616925 11836 consensus_queue.cc:1035] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:12.617141 11993 consensus_queue.cc:1035] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:12.625052 11995 mvcc.cc:205] Tried to move back new op lower bound from 7315383142870650880 to 7315383142810857472. Current Snapshot: MvccSnapshot[applied={T|T < 7315383142870650880 or (T in {7315383142870650880})}]
I20260806 02:10:12.627306 11996 mvcc.cc:205] Tried to move back new op lower bound from 7315383142870650880 to 7315383142810857472. Current Snapshot: MvccSnapshot[applied={T|T < 7315383142870650880}]
I20260806 02:10:12.631472 11491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:12.631641 11491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:12.633878 11491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Sending DeleteTablet for 3 replicas of tablet 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:12.634562 11620 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a401dc5811466da2e418e0cbaa2511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:12 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:49112
I20260806 02:10:12.634611 11759 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a401dc5811466da2e418e0cbaa2511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:12 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:34752
I20260806 02:10:12.634634 11903 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a401dc5811466da2e418e0cbaa2511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:12 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260806 02:10:12.634785 11837 tablet_replica.cc:330] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: stopping tablet replica
I20260806 02:10:12.634915 11837 raft_consensus.cc:2229] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:10:12.635165 11837 raft_consensus.cc:2258] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:12.635635 12004 tablet_replica.cc:330] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: stopping tablet replica
I20260806 02:10:12.635946 12004 raft_consensus.cc:2229] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:12.636229 11837 ts_tablet_manager.cc:1905] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:12.636240 12004 raft_consensus.cc:2258] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:12.636619 12005 tablet_replica.cc:330] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: stopping tablet replica
I20260806 02:10:12.636878 12005 raft_consensus.cc:2229] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:12.637138 12005 raft_consensus.cc:2258] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:12.637720 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11532
I20260806 02:10:12.641754 12005 ts_tablet_manager.cc:1905] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:12.642290 11837 ts_tablet_manager.cc:1918] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:12.642478 11837 log.cc:1196] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:12.642958 11837 ts_tablet_manager.cc:1939] T 68a401dc5811466da2e418e0cbaa2511 P e8baffcab7154bb19f6f4f87c57f7783: Deleting consensus metadata
I20260806 02:10:12.644424 12005 ts_tablet_manager.cc:1918] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:12.644537 12005 log.cc:1196] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:12.644873 12005 ts_tablet_manager.cc:1939] T 68a401dc5811466da2e418e0cbaa2511 P a893c21fdb824275801a830ce8e27f5d: Deleting consensus metadata
I20260806 02:10:12.645967 11477 catalog_manager.cc:4841] TS e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): tablet 68a401dc5811466da2e418e0cbaa2511 (table test-table [id=c3e7286e460540659bc5955f9b35b97a]) successfully deleted
I20260806 02:10:12.646206 11476 catalog_manager.cc:4841] TS a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): tablet 68a401dc5811466da2e418e0cbaa2511 (table test-table [id=c3e7286e460540659bc5955f9b35b97a]) successfully deleted
W20260806 02:10:12.652714 11478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:46563 (error 108)
I20260806 02:10:12.653522 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11694
I20260806 02:10:12.655211 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:12.653700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:46563 (local address 127.0.0.1:49118)
0806 02:10:12.655018 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:12.655104 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:46563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1210}
W20260806 02:10:12.655411 11478 catalog_manager.cc:4584] TS 3a5ba8107453471e895d477ca3ddf755 (127.28.21.1:46563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68a401dc5811466da2e418e0cbaa2511: Network error: Client connection negotiation failed: client connection to 127.28.21.1:46563: connect: Connection refused (error 111)
I20260806 02:10:12.662794 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11831
I20260806 02:10:12.669661 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 11457
I20260806 02:10:12.677054 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--webserver_interface=127.28.21.62
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:12.788935 12007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:12.789194 12007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:12.789273 12007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:12.793357 12007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:12.793416 12007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:12.793437 12007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:12.793453 12007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:12.793504 12007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:12.798844 12007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:12.799885 12007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:12.800922 12007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:10:12.809393 12007 server_base.cc:1034] running on GCE node
W20260806 02:10:12.809350 12014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.809373 12013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:12.809345 12016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:12.810065 12007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:12.811192 12007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:12.812350 12007 hybrid_clock.cc:648] HybridClock initialized: now 1785982212812319 us; error 44 us; skew 500 ppm
Aug 06 02:10:12 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:12.815325 12007 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:12.816522 12007 webserver.cc:460] Webserver started at http://127.28.21.62:37513/ using document root <none> and password file <none>
I20260806 02:10:12.817257 12007 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:12.817363 12007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:12.821235 12007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:12.823442 12023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:12.824656 12007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:12.824781 12007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "d2a2a831d9c042e99d276f75cca1637b"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "0fa680357c446686e8591aa2e6db447e"
server_key_iv: "928aa2219e916954415f70e13b5a63e2"
server_key_version: "encryptionkey@0"
I20260806 02:10:12.825305 12007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:12.838901 12007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:12.841585 12007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:12.841790 12007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:12.848945 12007 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:38089
I20260806 02:10:12.849253 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:38089 every 8 connection(s)
I20260806 02:10:12.850052 12007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:12.853436 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12007
I20260806 02:10:12.853870 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:46563
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=37275
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:12.854348 12078 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:10:12.855974 12078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap starting.
I20260806 02:10:12.873605 12078 log.cc:824] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:12.884737 12078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:12.885135 12078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b: Bootstrap complete.
I20260806 02:10:12.889232 12078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:12.889653 12078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Initialized, Role: FOLLOWER
I20260806 02:10:12.890126 12078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:12.890278 12078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:12.890365 12078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:12.890506 12078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:10:12.892328 12078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:12.892653 12078 leader_election.cc:302] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a2a831d9c042e99d276f75cca1637b; no voters:
I20260806 02:10:12.892968 12078 leader_election.cc:288] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:10:12.893105 12083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:10:12.893343 12083 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [term 3 LEADER]: Becoming Leader. State: Replica: d2a2a831d9c042e99d276f75cca1637b, State: Running, Role: LEADER
I20260806 02:10:12.893656 12083 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } }
I20260806 02:10:12.894143 12078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:12.895009 12084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:12.895130 12084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:12.895800 12085 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a2a831d9c042e99d276f75cca1637b. Latest consensus state: current_term: 3 leader_uuid: "d2a2a831d9c042e99d276f75cca1637b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a2a831d9c042e99d276f75cca1637b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 38089 } } }
I20260806 02:10:12.895891 12085 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a2a831d9c042e99d276f75cca1637b [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:12.896282 12092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:12.901937 12092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:12.902571 12092 catalog_manager.cc:1259] Loaded cluster ID: c48c7b1bebc642dbb594a7af0731840f
I20260806 02:10:12.902631 12092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:10:12.903702 12092 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2689bf9ad google::LogMessage::Fail() at ??:0
@ 0x7ff2689c288a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2689bf4de google::LogMessage::Flush() at ??:0
@ 0x7ff2689c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2732fefdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff2732ffd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2732fc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff273340640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff26ac113d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff26ac11c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff26ac133f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff26ac02512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff26c2466db start_thread at ??:0
@ 0x7ff2674bd61f clone at ??:0
W20260806 02:10:12.999686 12080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:12.999943 12080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:12.999995 12080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:13.004036 12080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:13.004096 12080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:13.004154 12080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:13.009604 12080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:46563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=37275
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:13.010848 12080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:13.011931 12080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:13.021833 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:13.021854 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:13.022119 12080 server_base.cc:1034] running on GCE node
W20260806 02:10:13.021850 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:13.022826 12080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:13.023651 12080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:13.024857 12080 hybrid_clock.cc:648] HybridClock initialized: now 1785982213024832 us; error 33 us; skew 500 ppm
Aug 06 02:10:13 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:13.028362 12080 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:13.029568 12080 webserver.cc:460] Webserver started at http://127.28.21.1:37275/ using document root <none> and password file <none>
I20260806 02:10:13.030308 12080 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:13.030438 12080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:13.035694 12080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:10:13.038445 12134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.040082 12080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:13.040239 12080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "3a5ba8107453471e895d477ca3ddf755"
format_stamp: "Formatted at 2026-08-06 02:10:10 on dist-test-slave-gp6n"
server_key: "c0b95e59c5eb545d2215e7ae9081ff92"
server_key_iv: "210c71616aa874371644155861660f82"
server_key_version: "encryptionkey@0"
I20260806 02:10:13.041056 12080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:13.069036 12080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:13.072917 12080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:13.073091 12080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:13.073681 12080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:13.074872 12141 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:10:13.079388 12080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:10:13.079496 12080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 02:10:13.079573 12080 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:10:13.081333 12080 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:10:13.081393 12080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:10:13.081692 12141 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Bootstrap starting.
I20260806 02:10:13.089080 12141 log.cc:824] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:13.091944 12080 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:46563
I20260806 02:10:13.092039 12256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:46563 every 8 connection(s)
I20260806 02:10:13.092826 12080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:13.097646 12141 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:13.098052 12141 tablet_bootstrap.cc:492] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Bootstrap complete.
I20260806 02:10:13.098384 12141 ts_tablet_manager.cc:1397] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20260806 02:10:13.100591 12141 raft_consensus.cc:348] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:13.100975 12141 raft_consensus.cc:729] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5ba8107453471e895d477ca3ddf755, State: Initialized, Role: FOLLOWER
I20260806 02:10:13.101408 12141 consensus_queue.cc:260] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:13.102125 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12080
I20260806 02:10:13.102421 12141 ts_tablet_manager.cc:1428] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260806 02:10:13.102626 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:40977
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=40221
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:13.161799 12259 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:10:13.094906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:46243)
0806 02:10:13.095396 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.095447 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:13.096235 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:13.160791 (+ 64556us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":149,"threads_started":1}
W20260806 02:10:13.162482 12257 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:38089 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:38089: Client connection negotiation failed: client connection to 127.28.21.62:38089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:10:13.163252 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.162955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:50071)
0806 02:10:13.163101 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.163181 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:10:13.163923 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.163680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:58653)
0806 02:10:13.163799 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.163852 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:10:13.164125 12257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:10:13.230923 12261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:13.231168 12261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:13.231210 12261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:13.235100 12261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:13.235183 12261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:13.235247 12261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:13.240262 12261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:40977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=40221
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:13.241173 12261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:13.242118 12261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:13.251717 12266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:13.251784 12261 server_base.cc:1034] running on GCE node
W20260806 02:10:13.251739 12267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:13.251940 12269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:13.252290 12261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:13.252880 12261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:13.254037 12261 hybrid_clock.cc:648] HybridClock initialized: now 1785982213254025 us; error 24 us; skew 500 ppm
Aug 06 02:10:13 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:13.256987 12261 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:13.257854 12261 webserver.cc:460] Webserver started at http://127.28.21.2:40221/ using document root <none> and password file <none>
I20260806 02:10:13.258450 12261 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:13.258528 12261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:13.262123 12261 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:13.264132 12277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.265331 12261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:10:13.265468 12261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "e8baffcab7154bb19f6f4f87c57f7783"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "519db1558d6f15be1b6658809a4b432c"
server_key_iv: "f38c09933b3f5694a13910cf24e7af2b"
server_key_version: "encryptionkey@0"
I20260806 02:10:13.265995 12261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:13.293277 12261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:13.296285 12261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:13.296506 12261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:13.297137 12261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:13.297984 12261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:13.298053 12261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.298130 12261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:13.298172 12261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.307809 12261 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:40977
I20260806 02:10:13.307837 12394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:40977 every 8 connection(s)
I20260806 02:10:13.309043 12261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:13.310717 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12261
I20260806 02:10:13.311116 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:37637
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:38089
--webserver_port=35615
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:13.313499 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.311477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:37761)
0806 02:10:13.312039 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.312630 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
W20260806 02:10:13.313977 12395 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:38089 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:38089: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
I20260806 02:10:13.314616 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.314380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:34897)
0806 02:10:13.314519 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.314564 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:10:13.315258 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.315081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:37395)
0806 02:10:13.315172 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.315212 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 02:10:13.315467 12395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:10:13.423764 12398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:13.424013 12398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:13.424081 12398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:13.428174 12398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:13.428228 12398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:13.428324 12398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:13.433518 12398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=35615
--tserver_master_addrs=127.28.21.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:13.434568 12398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:13.435575 12398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:13.445209 12403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:13.445281 12406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:13.445282 12404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:13.445528 12398 server_base.cc:1034] running on GCE node
I20260806 02:10:13.445935 12398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:13.446611 12398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:13.447789 12398 hybrid_clock.cc:648] HybridClock initialized: now 1785982213447766 us; error 37 us; skew 500 ppm
Aug 06 02:10:13 dist-test-slave-gp6n krb5kdc[10878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:13.450631 12398 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:13.451465 12398 webserver.cc:460] Webserver started at http://127.28.21.3:35615/ using document root <none> and password file <none>
I20260806 02:10:13.452013 12398 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:13.452103 12398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:13.455775 12398 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:13.457729 12413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.458971 12398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:13.459100 12398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "a893c21fdb824275801a830ce8e27f5d"
format_stamp: "Formatted at 2026-08-06 02:10:11 on dist-test-slave-gp6n"
server_key: "d5156e6e781fbe671efbbbbaaeb78fab"
server_key_iv: "afd0a558fa546f5090aa78b127c82722"
server_key_version: "encryptionkey@0"
I20260806 02:10:13.459635 12398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:13.500515 12398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:13.503458 12398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:13.503672 12398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:13.504285 12398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:13.505117 12398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:13.505185 12398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.505256 12398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:13.505301 12398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:13.514180 12398 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:37637
I20260806 02:10:13.514214 12529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:37637 every 8 connection(s)
I20260806 02:10:13.515226 12398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:13.518919 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.517326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:40727)
0806 02:10:13.517761 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.518197 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260806 02:10:13.519250 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12398
W20260806 02:10:13.519407 12530 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:38089 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:38089: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
I20260806 02:10:13.520076 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.519813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:48105)
0806 02:10:13.519920 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.520005 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:10:13.520751 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:13.520563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:48325)
0806 02:10:13.520657 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:13.520699 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:10:13.520912 12530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:10:14.165274 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.164773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:50601)
0806 02:10:14.165100 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:14.165159 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":79,"threads_started":1}
I20260806 02:10:14.316357 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.315987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:54685)
0806 02:10:14.316211 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:14.316262 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260806 02:10:14.521829 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.521423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:43057)
0806 02:10:14.521665 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:14.521719 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260806 02:10:14.860410 12537 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:14.860711 12537 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:14.861977 12537 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
Aug 06 02:10:14 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.3@KRBTEST.COM
Aug 06 02:10:14 dist-test-slave-gp6n krb5kdc[10878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.2@KRBTEST.COM
I20260806 02:10:14.881357 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.862305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:40977 (local address 127.28.21.1:37163)
0806 02:10:14.862723 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:14.862737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:14.862860 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:14.864688 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:14.864690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:14.864949 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:14.865397 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:14.865414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.866753 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.866757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:14.867642 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:14.867652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.868390 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.868394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:14.868640 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:14.873562 (+ 4922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:14.873591 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:14.875999 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:14.878510 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:14.878517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:14.878521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:14.878799 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:14.879210 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:14.879216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:14.879221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:14.879330 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:14.880277 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:14.880285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:14.880579 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 02:10:14.880788 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:10:14.881137 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":50,"spinlock_wait_cycles":27520,"thread_start_us":92,"threads_started":1}
I20260806 02:10:14.881570 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.862351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:52981 (local address 127.28.21.3:37637)
0806 02:10:14.862702 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:14.862734 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:14.862785 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:10:14.863633 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:14.863648 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:14.864052 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:14.864366 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:14.865585 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.866634 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.867776 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.868246 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.868440 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:14.875942 (+ 7502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:14.875948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:14.875965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:14.876000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:14.877898 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:14.878572 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:14.878583 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:14.878593 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:14.878751 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:14.879275 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:14.879281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:14.879286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:14.879964 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:14.880126 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:14.881090 (+ 964us) server_negotiation.cc:299] Negotiation successful
0806 02:10:14.881397 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260806 02:10:14.882083 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.862499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:37163 (local address 127.28.21.2:40977)
0806 02:10:14.862943 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:14.862956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:14.862990 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:10:14.863627 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:14.863634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:14.864059 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:14.864301 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:14.865586 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.866606 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.867777 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.868247 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.868435 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:14.876214 (+ 7779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:14.876220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:14.876232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:14.876264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:14.878337 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:14.878944 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:14.878951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:14.878958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:14.879106 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:14.879467 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:14.879473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:14.879474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:14.880098 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:14.880230 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:14.881685 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0806 02:10:14.881949 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":92,"threads_started":1}
I20260806 02:10:14.882608 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:14.862123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:37637 (local address 127.28.21.1:52981)
0806 02:10:14.862505 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:14.862519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:14.862632 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:14.864424 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:14.864429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:14.864924 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:14.865397 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:14.865414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.866757 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.866762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:14.867645 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:14.867655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:14.868390 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:14.868394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:14.868644 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:14.873586 (+ 4942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:14.873599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:14.875633 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:14.878054 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:14.878061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:14.878077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:14.878363 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:14.878886 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:14.878892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:14.878894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:14.879099 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:14.880104 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:14.880502 (+ 398us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:14.880579 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:10:14.880788 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:10:14.882398 (+ 1610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1506,"spinlock_wait_cycles":24832,"thread_start_us":105,"threads_started":1}
W20260806 02:10:14.885275 12137 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
W20260806 02:10:14.886531 12136 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:14.886662 12136 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755; no voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783
I20260806 02:10:14.886960 12537 raft_consensus.cc:2726] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:10:15.166519 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:15.166147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:46161)
0806 02:10:15.166314 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:15.166364 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:10:15.317512 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:15.317102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:35099)
0806 02:10:15.317339 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:15.317388 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260806 02:10:15.523000 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:15.522595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:33017)
0806 02:10:15.522857 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:15.522908 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260806 02:10:16.167712 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:16.167285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:36937)
0806 02:10:16.167548 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:16.167611 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260806 02:10:16.318722 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:16.318282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:58789)
0806 02:10:16.318570 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:16.318626 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260806 02:10:16.524113 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:16.523698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:60209)
0806 02:10:16.523946 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:16.524005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260806 02:10:16.641485 12547 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:16.641618 12547 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:16.642012 12547 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
W20260806 02:10:16.642818 12136 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
W20260806 02:10:16.642923 12137 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:16.643028 12137 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755; no voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783
I20260806 02:10:16.643172 12547 raft_consensus.cc:2726] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:10:17.168965 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:17.168547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:47687)
0806 02:10:17.168807 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:17.168866 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260806 02:10:17.319886 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:17.319440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:54311)
0806 02:10:17.319720 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:17.319774 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260806 02:10:17.525322 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:17.524907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:39123)
0806 02:10:17.525163 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:17.525223 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260806 02:10:18.170586 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:18.169987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:37265)
0806 02:10:18.170391 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:18.170474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":76,"threads_started":1}
I20260806 02:10:18.321203 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:18.320806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:57499)
0806 02:10:18.321061 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:18.321113 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260806 02:10:18.480379 12554 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:18.480589 12554 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:18.481029 12554 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
W20260806 02:10:18.481807 12137 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
W20260806 02:10:18.481913 12136 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:18.482045 12136 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755; no voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783
I20260806 02:10:18.482178 12554 raft_consensus.cc:2726] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:10:18.526710 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:18.526183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:59569)
0806 02:10:18.526530 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:18.526597 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260806 02:10:19.171949 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:19.171524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:36109)
0806 02:10:19.171780 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:19.171841 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260806 02:10:19.322379 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:19.321927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:34287)
0806 02:10:19.322202 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:19.322261 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260806 02:10:19.527931 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:19.527504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:43615)
0806 02:10:19.527766 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:19.527823 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260806 02:10:20.173187 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:20.172735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:36203)
0806 02:10:20.173020 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:20.173087 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260806 02:10:20.323578 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:20.323135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:50871)
0806 02:10:20.323414 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:20.323468 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260806 02:10:20.529160 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:20.528717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:45127)
0806 02:10:20.529010 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:20.529063 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
I20260806 02:10:21.174650 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:21.174045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:55821)
0806 02:10:21.174392 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:21.174538 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260806 02:10:21.324783 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:21.324359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:39547)
0806 02:10:21.324626 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:21.324686 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260806 02:10:21.530479 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:21.529974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:45677)
0806 02:10:21.530244 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:21.530302 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260806 02:10:22.175963 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:22.175540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:39581)
0806 02:10:22.175807 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:22.175863 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260806 02:10:22.325992 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:22.325521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:51491)
0806 02:10:22.325836 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:22.325896 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260806 02:10:22.531697 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:22.531237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:36327)
0806 02:10:22.531535 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:22.531594 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":137,"threads_started":1}
I20260806 02:10:22.884757 12568 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:22.884896 12568 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:22.885392 12568 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
W20260806 02:10:22.886212 12137 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
W20260806 02:10:22.886384 12136 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:22.886541 12136 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755; no voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783
I20260806 02:10:22.886715 12568 raft_consensus.cc:2726] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:10:23.177325 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:23.176898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:60809)
0806 02:10:23.177164 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:23.177222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260806 02:10:23.327381 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:23.326966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:38071)
0806 02:10:23.327217 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:23.327272 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260806 02:10:23.532912 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:23.532521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:52399)
0806 02:10:23.532753 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:23.532806 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260806 02:10:24.178658 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:24.178127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:50425)
0806 02:10:24.178482 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:24.178556 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260806 02:10:24.328640 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:24.328205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:46957)
0806 02:10:24.328480 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:24.328540 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260806 02:10:24.534214 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:24.533744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:57907)
0806 02:10:24.534032 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:24.534100 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260806 02:10:25.179944 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:25.179487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:49255)
0806 02:10:25.179776 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:25.179844 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260806 02:10:25.329918 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:25.329524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:59879)
0806 02:10:25.329763 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:25.329819 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260806 02:10:25.535539 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:25.535106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:41621)
0806 02:10:25.535375 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:25.535438 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260806 02:10:25.968509 12578 raft_consensus.cc:482] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:25.968648 12578 raft_consensus.cc:504] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5ba8107453471e895d477ca3ddf755" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 46563 } } peers { permanent_uuid: "a893c21fdb824275801a830ce8e27f5d" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 37637 } } peers { permanent_uuid: "e8baffcab7154bb19f6f4f87c57f7783" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 40977 } }
I20260806 02:10:25.969156 12578 leader_election.cc:288] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637), e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977)
W20260806 02:10:25.970129 12137 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8baffcab7154bb19f6f4f87c57f7783 (127.28.21.2:40977): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
W20260806 02:10:25.970243 12136 leader_election.cc:341] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a893c21fdb824275801a830ce8e27f5d (127.28.21.3:37637): Not found: Tablet not found: 68a401dc5811466da2e418e0cbaa2511
I20260806 02:10:25.970367 12136 leader_election.cc:302] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a5ba8107453471e895d477ca3ddf755; no voters: a893c21fdb824275801a830ce8e27f5d, e8baffcab7154bb19f6f4f87c57f7783
I20260806 02:10:25.970664 12578 raft_consensus.cc:2726] T 68a401dc5811466da2e418e0cbaa2511 P 3a5ba8107453471e895d477ca3ddf755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:10:26.181174 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:26.180743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:39279)
0806 02:10:26.181018 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:26.181076 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260806 02:10:26.331223 12580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:26.330769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:49639)
0806 02:10:26.331056 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:26.331119 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260806 02:10:26.536783 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:26.536368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:50787)
0806 02:10:26.536626 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:26.536682 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260806 02:10:27.182533 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:27.182057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:45131)
0806 02:10:27.182318 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:27.182371 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260806 02:10:27.332517 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:27.332065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:39357)
0806 02:10:27.332352 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:27.332412 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260806 02:10:27.538103 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:27.537636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:40775)
0806 02:10:27.537932 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:27.537998 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260806 02:10:28.183882 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:28.183465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.1:58521)
0806 02:10:28.183731 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:28.183787 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260806 02:10:28.333966 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:28.333537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.2:34845)
0806 02:10:28.333805 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:28.333857 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
I20260806 02:10:28.529065 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12080
I20260806 02:10:28.536890 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12261
I20260806 02:10:28.539485 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:28.539089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:38089 (local address 127.28.21.3:35361)
0806 02:10:28.539328 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:28.539391 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260806 02:10:28.543386 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12398
2026-08-06T02:10:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19260 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:10:28 dist-test-slave-gp6n krb5kdc[12590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:10:28 dist-test-slave-gp6n krb5kdc[12590](info): set up 2 sockets
Aug 06 02:10:28 dist-test-slave-gp6n krb5kdc[12590](info): commencing operation
krb5kdc: starting...
W20260806 02:10:29.613875 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:10:29 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:10:29.742069 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:29.855680 12603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:29.855952 12603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:29.855993 12603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:29.860127 12603 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:10:29.860188 12603 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:10:29.860203 12603 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:10:29.860217 12603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:29.860236 12603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:29.860277 12603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:29.860314 12603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:29.860335 12603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:29.865608 12603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:29.866799 12603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:29.867878 12603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:29.877123 12608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:29.877194 12611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:29.877511 12609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:29.878541 12603 server_base.cc:1034] running on GCE node
I20260806 02:10:29.879300 12603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:29.880436 12603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:29.881644 12603 hybrid_clock.cc:648] HybridClock initialized: now 1785982229881616 us; error 52 us; skew 500 ppm
Aug 06 02:10:29 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:29.889972 12603 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:29.891074 12603 webserver.cc:460] Webserver started at http://127.28.21.62:45591/ using document root <none> and password file <none>
I20260806 02:10:29.891691 12603 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:29.891788 12603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:29.892035 12603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:29.894176 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "e7431e4ed6554737b486ff8208ccc2f0"
format_stamp: "Formatted at 2026-08-06 02:10:29 on dist-test-slave-gp6n"
server_key: "64501afab4ce29ba8a1b45602556d243"
server_key_iv: "cb60aa982b24547f0cafad8484c6ae15"
server_key_version: "encryptionkey@0"
I20260806 02:10:29.894801 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "e7431e4ed6554737b486ff8208ccc2f0"
format_stamp: "Formatted at 2026-08-06 02:10:29 on dist-test-slave-gp6n"
server_key: "64501afab4ce29ba8a1b45602556d243"
server_key_iv: "cb60aa982b24547f0cafad8484c6ae15"
server_key_version: "encryptionkey@0"
I20260806 02:10:29.898618 12603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:10:29.901000 12618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:29.902236 12603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:29.902451 12603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "e7431e4ed6554737b486ff8208ccc2f0"
format_stamp: "Formatted at 2026-08-06 02:10:29 on dist-test-slave-gp6n"
server_key: "64501afab4ce29ba8a1b45602556d243"
server_key_iv: "cb60aa982b24547f0cafad8484c6ae15"
server_key_version: "encryptionkey@0"
I20260806 02:10:29.902578 12603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:29.935439 12603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:29.938551 12603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:29.938768 12603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:29.946208 12603 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:44115
I20260806 02:10:29.946533 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:44115 every 8 connection(s)
I20260806 02:10:29.947261 12603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:29.949515 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12603
I20260806 02:10:29.949689 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:10:29.950006 28756 external_mini_cluster.cc:1428] Setting key 4e7a30d09ee40390a0316f4a0f7cf869
I20260806 02:10:29.950244 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:29.957653 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap starting.
Aug 06 02:10:29 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:29.960229 12677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:29.961180 12677 log.cc:824] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:29.963624 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: No bootstrap required, opened a new log
I20260806 02:10:29.965472 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:29.951500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.28.21.62:44115)
0806 02:10:29.951947 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:29.951966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:29.952010 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:10:29.953090 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:29.953116 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:29.953546 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:29.953939 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:29.954661 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:29.956144 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:29.957001 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:29.957686 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:29.957857 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:29.960598 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:29.960604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:29.960621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:29.960651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:29.962611 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:29.963131 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:29.963144 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:29.963148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:29.963260 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:29.963563 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:29.963565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:29.963566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:29.964016 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:29.964161 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:29.964446 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:10:29.964728 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
I20260806 02:10:29.967941 12677 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:29.968237 12677 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:29.968307 12677 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Initialized, Role: FOLLOWER
I20260806 02:10:29.968775 12677 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:29.968943 12677 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:29.969017 12677 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:29.969122 12677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:29.970072 12677 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:29.970468 12677 leader_election.cc:302] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7431e4ed6554737b486ff8208ccc2f0; no voters:
I20260806 02:10:29.970890 12677 leader_election.cc:288] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:10:29.971045 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:29.971267 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 LEADER]: Becoming Leader. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Running, Role: LEADER
I20260806 02:10:29.971626 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:29.972141 12677 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:29.973402 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431e4ed6554737b486ff8208ccc2f0. Latest consensus state: current_term: 1 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:10:29.973409 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:10:29.973544 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:29.973547 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:29.973966 12693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:29.977146 12693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:29.983201 12693 catalog_manager.cc:1347] Generated new cluster ID: 6c89c2a8f8874f47a64b8bcf68d214b8
I20260806 02:10:29.983354 12693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:29.993541 12693 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:10:29.994925 12693 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:30.002002 12693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Generated new TSK 0
I20260806 02:10:30.002977 12693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:10:30.045428 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:30.162418 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:30.162689 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:30.162734 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:30.166801 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:30.166865 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:30.166963 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:30.172639 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:30.173813 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:30.174883 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:30.184842 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:30.184886 12705 server_base.cc:1034] running on GCE node
W20260806 02:10:30.184906 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:30.184885 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:30.185564 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:30.186511 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:30.187733 12705 hybrid_clock.cc:648] HybridClock initialized: now 1785982230187701 us; error 46 us; skew 500 ppm
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:30.190703 12705 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:30.191570 12705 webserver.cc:460] Webserver started at http://127.28.21.1:44099/ using document root <none> and password file <none>
I20260806 02:10:30.192135 12705 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:30.192196 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:30.192443 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:30.194541 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "5f7eba7b375c463da0b4dbba11e4f786"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "3158a4eceeb96e3eac2c7fa001e7739c"
server_key_iv: "c0b35319f63cdd9fca377f54ac772c5c"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.195103 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "5f7eba7b375c463da0b4dbba11e4f786"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "3158a4eceeb96e3eac2c7fa001e7739c"
server_key_iv: "c0b35319f63cdd9fca377f54ac772c5c"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.198794 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:30.201471 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.202769 12705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:30.202914 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "5f7eba7b375c463da0b4dbba11e4f786"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "3158a4eceeb96e3eac2c7fa001e7739c"
server_key_iv: "c0b35319f63cdd9fca377f54ac772c5c"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.203016 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:30.231729 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:30.235226 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:30.235423 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:30.236034 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:30.236845 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:30.236900 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.236969 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:30.236999 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.245982 12705 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:42685
I20260806 02:10:30.246017 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:42685 every 8 connection(s)
I20260806 02:10:30.246958 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:30.253062 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12705
I20260806 02:10:30.253216 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:10:30.253535 28756 external_mini_cluster.cc:1428] Setting key 1b728ec6c49344148606558a2bcd59b6
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:30.262470 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.248991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:33427 (local address 127.28.21.62:44115)
0806 02:10:30.249142 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.249148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.250018 (+ 870us) server_negotiation.cc:407] Connection header received
0806 02:10:30.251435 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.251438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.251489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.251572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:30.252964 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.253752 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.255381 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.255651 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.255776 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.258767 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:30.258770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:30.258772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:30.258794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:30.260354 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.260887 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.260895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.260896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.260975 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.261326 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.261331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.261337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.261513 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:30.261621 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.262129 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.262259 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:10:30.263294 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.249382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:33427)
0806 02:10:30.249852 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.249896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.251177 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.251782 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.251791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.252193 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:30.252758 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.252773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.253899 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.253903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.255231 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.255241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.255816 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.255820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.256014 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.256678 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:30.256696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:30.258590 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:30.260479 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.260485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.260499 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.260767 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.261065 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.261069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.261071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.261217 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.261621 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:30.261628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:30.261888 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.262071 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.262398 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
I20260806 02:10:30.264595 12840 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:30.264812 12840 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:30.265267 12840 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:10:30.266953 12636 ts_manager.cc:194] Registered new tserver with Master: 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685)
I20260806 02:10:30.268250 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:33427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:10:30.283605 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:30.397501 12845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:30.397778 12845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:30.397822 12845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:30.401959 12845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:30.402020 12845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:30.402112 12845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:30.407624 12845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:30.408743 12845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:30.409888 12845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:30.420117 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:30.420117 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:30.420264 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:30.420452 12845 server_base.cc:1034] running on GCE node
I20260806 02:10:30.420961 12845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:30.421626 12845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:30.422820 12845 hybrid_clock.cc:648] HybridClock initialized: now 1785982230422781 us; error 54 us; skew 500 ppm
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:30.425704 12845 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:30.426694 12845 webserver.cc:460] Webserver started at http://127.28.21.2:40287/ using document root <none> and password file <none>
I20260806 02:10:30.427227 12845 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:30.427284 12845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:30.427446 12845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:30.429441 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "a318f3b366674118a30bef8093f3bc90"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "1033c164230ac8e92e7e2022522d306b"
server_key_iv: "25e917ba5efc16183154fd35cd2b1923"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.430042 12845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "a318f3b366674118a30bef8093f3bc90"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "1033c164230ac8e92e7e2022522d306b"
server_key_iv: "25e917ba5efc16183154fd35cd2b1923"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.433701 12845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:10:30.436089 12862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.437383 12845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:10:30.437479 12845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a318f3b366674118a30bef8093f3bc90"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "1033c164230ac8e92e7e2022522d306b"
server_key_iv: "25e917ba5efc16183154fd35cd2b1923"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.437567 12845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:30.454545 12845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:30.457604 12845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:30.457759 12845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:30.458303 12845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:30.459167 12845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:30.459214 12845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.459254 12845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:30.459275 12845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.468394 12845 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:36395
I20260806 02:10:30.468456 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:36395 every 8 connection(s)
I20260806 02:10:30.469308 12845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:30.470731 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12845
I20260806 02:10:30.470847 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:10:30.471159 28756 external_mini_cluster.cc:1428] Setting key 3a19eb4e0920e2c304540a0878071a41
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:10:30.483773 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.471201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:59761 (local address 127.28.21.62:44115)
0806 02:10:30.471348 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.471352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.472222 (+ 870us) server_negotiation.cc:407] Connection header received
0806 02:10:30.473743 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.473746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.473798 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.473885 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:30.475190 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.475736 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.476874 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.477130 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.477236 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.480120 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:30.480126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:30.480129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:30.480158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:30.481785 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.482298 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.482300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.482302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.482378 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.482769 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.482772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.482773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.482951 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:30.483030 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.483489 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.483651 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:10:30.484635 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.471617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:59761)
0806 02:10:30.472062 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.472114 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.473515 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.474059 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.474065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.474569 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:30.475048 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.475061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.475871 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.475874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.476747 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.476757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.477283 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.477287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.477487 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.478129 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:30.478146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:30.479944 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:30.481894 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.481899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.481909 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.482149 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.482549 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.482551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.482553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.482687 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.483040 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:30.483044 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:30.483262 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.483411 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.483714 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260806 02:10:30.485913 12984 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:30.486212 12984 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:30.486846 12984 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
I20260806 02:10:30.488052 12636 ts_manager.cc:194] Registered new tserver with Master: a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
I20260806 02:10:30.489035 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:59761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:10:30.501475 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:30.615447 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:30.615770 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:30.615847 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:30.620064 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:30.620118 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:30.620177 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:30.625214 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:30.626183 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:30.627326 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:30.636950 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:30.637025 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:30.636950 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:30.637106 12989 server_base.cc:1034] running on GCE node
I20260806 02:10:30.637800 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:30.638475 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:30.639657 12989 hybrid_clock.cc:648] HybridClock initialized: now 1785982230639623 us; error 48 us; skew 500 ppm
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:30.642650 12989 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:30.643451 12989 webserver.cc:460] Webserver started at http://127.28.21.3:43187/ using document root <none> and password file <none>
I20260806 02:10:30.643992 12989 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:30.644052 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:30.644218 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:10:30.646204 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "b6beb872c50643f4909d395beba78c4c"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "5f992fa0723d3874628df9e8349493bb"
server_key_iv: "b0ab453a665a22097abde9c5362fc7fc"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.646801 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "b6beb872c50643f4909d395beba78c4c"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "5f992fa0723d3874628df9e8349493bb"
server_key_iv: "b0ab453a665a22097abde9c5362fc7fc"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.650478 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:30.652861 13006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.654055 12989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:30.654174 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "b6beb872c50643f4909d395beba78c4c"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "5f992fa0723d3874628df9e8349493bb"
server_key_iv: "b0ab453a665a22097abde9c5362fc7fc"
server_key_version: "encryptionkey@0"
I20260806 02:10:30.654265 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:30.672416 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:30.675930 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:30.676085 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:30.676625 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:30.677459 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:30.677508 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.677556 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:30.677572 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:30.686638 12989 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:44907
I20260806 02:10:30.686731 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:44907 every 8 connection(s)
I20260806 02:10:30.687613 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
Aug 06 02:10:30 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:30.697777 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 12989
I20260806 02:10:30.697924 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:10:30.698235 28756 external_mini_cluster.cc:1428] Setting key 75b3058a5817125e48a7d3c21ebeb991
I20260806 02:10:30.701417 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.689492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59003 (local address 127.28.21.62:44115)
0806 02:10:30.689668 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.689671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.690540 (+ 869us) server_negotiation.cc:407] Connection header received
0806 02:10:30.691875 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.691879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.691937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.692042 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:30.693173 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.693720 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.694940 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.695173 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.695285 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.697726 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:30.697730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:30.697733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:30.697763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:30.699420 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.699997 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.700000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.700001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.700095 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.700469 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.700472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.700475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.700668 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:30.700748 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.701203 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.701303 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:10:30.702216 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.689879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:59003)
0806 02:10:30.690331 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.690372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.691656 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.692209 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.692215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.692566 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:30.693015 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.693027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.693843 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.693848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.694816 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.694825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.695326 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.695330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.695500 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.696102 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:30.696118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:30.697565 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:30.699534 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.699540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.699554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.699890 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.700187 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:30.700190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:30.700192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:30.700380 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:30.700760 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:30.700765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:30.700980 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.701128 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.701388 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260806 02:10:30.703311 13129 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:30.703547 13129 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:30.703997 13129 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
I20260806 02:10:30.704931 12636 ts_manager.cc:194] Registered new tserver with Master: b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907)
I20260806 02:10:30.705472 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:59003
I20260806 02:10:30.712198 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:30.721895 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.714273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.28.21.62:44115)
0806 02:10:30.714481 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.714487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.714664 (+ 177us) server_negotiation.cc:407] Connection header received
0806 02:10:30.714822 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.714828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.714885 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.714972 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:30.715792 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.716304 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.717162 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.717408 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.717486 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.718862 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:30.718865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:30.718867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:30.718891 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:30.720460 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.720839 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.720842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.720843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.720918 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:30.721157 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:30.721160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:30.721161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:30.721364 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:30.721497 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.721693 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.721782 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:10:30.724761 12636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:10:30.727885 12636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:10:30.742713 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.735978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.28.21.1:42685)
0806 02:10:30.736382 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.736398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.736769 (+ 371us) server_negotiation.cc:407] Connection header received
0806 02:10:30.736994 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.737000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.737238 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.737370 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:30.737938 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739179 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.741199 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.742020 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.742130 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.742274 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.742457 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.742568 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":123,"threads_started":1}
I20260806 02:10:30.742702 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.735844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.0.0.1:49282)
0806 02:10:30.736618 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.736668 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.736827 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.737590 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.737595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.737618 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:30.737755 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.737766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739325 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.739331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.741067 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.741079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.742164 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.742167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.742247 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.742276 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.742396 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.742556 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":150,"thread_start_us":89,"threads_started":1}
I20260806 02:10:30.743039 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.736634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.28.21.3:44907)
0806 02:10:30.737374 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.737386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.737407 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:10:30.737476 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.737481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.737669 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.737814 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:30.738206 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739328 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.741077 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.742020 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.742138 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.742274 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.742770 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.742870 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":147,"threads_started":1}
I20260806 02:10:30.743228 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.736211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:44907 (local address 127.0.0.1:56054)
0806 02:10:30.737025 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.737039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.737112 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.737983 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.737986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.738010 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:30.738090 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.738094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739494 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.739497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.740924 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.740934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.742343 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.742346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.742468 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.742500 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.742574 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.742640 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":45,"thread_start_us":81,"threads_started":1}
I20260806 02:10:30.745177 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.735931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36395 (local address 127.0.0.1:54506)
0806 02:10:30.736671 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.736683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.736806 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.737665 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.737668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.737687 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:30.737805 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.737813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739439 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.739445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.741067 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.741079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.744779 (+ 3700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.744784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.744876 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.744898 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.744986 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.745051 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":221,"thread_start_us":108,"threads_started":1}
I20260806 02:10:30.745240 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.736634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.28.21.2:36395)
0806 02:10:30.737180 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.737193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.737215 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:10:30.737293 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.737299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.737507 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.737691 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:30.738516 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.739286 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.742337 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.744645 (+ 2308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.744757 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.744887 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.745013 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.745098 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":120,"threads_started":1}
I20260806 02:10:30.746083 12770 tablet_service.cc:1467] Processing CreateTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 (DEFAULT_TABLE table=test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:30.746117 13056 tablet_service.cc:1467] Processing CreateTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 (DEFAULT_TABLE table=test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:30.747164 13056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653080f0c8d34e2dbaed018b18a1e9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:30.747210 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653080f0c8d34e2dbaed018b18a1e9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:30.748210 12916 tablet_service.cc:1467] Processing CreateTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 (DEFAULT_TABLE table=test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:30.749186 12916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653080f0c8d34e2dbaed018b18a1e9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:30.753679 13146 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Bootstrap starting.
I20260806 02:10:30.755086 13147 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap starting.
I20260806 02:10:30.755867 13148 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Bootstrap starting.
I20260806 02:10:30.756534 13146 tablet_bootstrap.cc:654] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:30.757115 13147 tablet_bootstrap.cc:654] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:30.757449 13146 log.cc:824] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:30.758350 13147 log.cc:824] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:30.758549 13148 tablet_bootstrap.cc:654] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:30.759344 13148 log.cc:824] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:30.759686 13146 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: No bootstrap required, opened a new log
I20260806 02:10:30.759888 13146 ts_tablet_manager.cc:1397] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:30.760226 13147 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: No bootstrap required, opened a new log
I20260806 02:10:30.760421 13147 ts_tablet_manager.cc:1397] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:10:30.761315 13148 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: No bootstrap required, opened a new log
I20260806 02:10:30.761492 13148 ts_tablet_manager.cc:1397] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:30.763411 13147 raft_consensus.cc:348] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.763653 13147 raft_consensus.cc:374] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:30.763715 13147 raft_consensus.cc:729] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7eba7b375c463da0b4dbba11e4f786, State: Initialized, Role: FOLLOWER
I20260806 02:10:30.763839 13146 raft_consensus.cc:348] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.764117 13146 raft_consensus.cc:374] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:30.764183 13146 raft_consensus.cc:729] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6beb872c50643f4909d395beba78c4c, State: Initialized, Role: FOLLOWER
I20260806 02:10:30.764417 13147 consensus_queue.cc:260] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.764616 13146 consensus_queue.cc:260] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.765242 13148 raft_consensus.cc:348] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.765429 13147 ts_tablet_manager.cc:1428] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:10:30.765488 13148 raft_consensus.cc:374] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:30.765542 13148 raft_consensus.cc:729] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a318f3b366674118a30bef8093f3bc90, State: Initialized, Role: FOLLOWER
I20260806 02:10:30.765592 13146 ts_tablet_manager.cc:1428] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:10:30.765780 12840 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
I20260806 02:10:30.765920 13129 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
I20260806 02:10:30.765949 13148 consensus_queue.cc:260] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.766595 12984 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
I20260806 02:10:30.766842 13148 ts_tablet_manager.cc:1428] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:10:30.801263 13154 raft_consensus.cc:482] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:30.801489 13154 raft_consensus.cc:504] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.802677 13154 leader_election.cc:288] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685), b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907)
I20260806 02:10:30.806980 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.803034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:44907 (local address 127.28.21.2:33429)
0806 02:10:30.803278 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.803296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.803396 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.803748 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.803751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.803767 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:30.803918 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.803926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.804783 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.804789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.805743 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.805753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.806512 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.806518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.806637 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.806664 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.806777 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.806854 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":40}
I20260806 02:10:30.806999 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.803275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:33429 (local address 127.28.21.3:44907)
0806 02:10:30.803473 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.803478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.803493 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:10:30.803568 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.803573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.803626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.803708 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:30.804053 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.804656 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.805903 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.806392 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.806533 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.806599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.806802 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.806882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:10:30.807056 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.803146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:42345 (local address 127.28.21.1:42685)
0806 02:10:30.803291 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.803296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.803615 (+ 319us) server_negotiation.cc:407] Connection header received
0806 02:10:30.803716 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.803721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.803789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.803898 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:30.804204 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.804695 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.805831 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.806391 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.806565 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.806632 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.806855 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.806924 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:10:30.807648 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.803040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.28.21.2:42345)
0806 02:10:30.803481 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.803496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:30.803609 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:30.803915 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:30.803917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:30.803931 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:30.804083 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.804090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.804833 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.804835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.805704 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:30.805713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.806565 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.806569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:30.806692 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.806723 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:10:30.807415 (+ 692us) client_negotiation.cc:240] Negotiation successful
0806 02:10:30.807492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":44,"thread_start_us":136,"threads_started":1}
I20260806 02:10:30.807610 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653080f0c8d34e2dbaed018b18a1e9c3" candidate_uuid: "a318f3b366674118a30bef8093f3bc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6beb872c50643f4909d395beba78c4c" is_pre_election: true
I20260806 02:10:30.807938 13079 raft_consensus.cc:2454] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a318f3b366674118a30bef8093f3bc90 in term 0.
I20260806 02:10:30.808329 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653080f0c8d34e2dbaed018b18a1e9c3" candidate_uuid: "a318f3b366674118a30bef8093f3bc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7eba7b375c463da0b4dbba11e4f786" is_pre_election: true
I20260806 02:10:30.808470 12865 leader_election.cc:302] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c; no voters:
I20260806 02:10:30.808651 12792 raft_consensus.cc:2454] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a318f3b366674118a30bef8093f3bc90 in term 0.
I20260806 02:10:30.808795 13154 raft_consensus.cc:2781] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:30.808905 13154 raft_consensus.cc:482] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:30.808966 13154 raft_consensus.cc:3037] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:30.810078 13154 raft_consensus.cc:504] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.810503 13154 leader_election.cc:288] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [CANDIDATE]: Term 1 election: Requested vote from peers 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685), b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907)
I20260806 02:10:30.810820 13079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653080f0c8d34e2dbaed018b18a1e9c3" candidate_uuid: "a318f3b366674118a30bef8093f3bc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6beb872c50643f4909d395beba78c4c"
I20260806 02:10:30.810817 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653080f0c8d34e2dbaed018b18a1e9c3" candidate_uuid: "a318f3b366674118a30bef8093f3bc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7eba7b375c463da0b4dbba11e4f786"
I20260806 02:10:30.810936 12792 raft_consensus.cc:3037] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:30.810936 13079 raft_consensus.cc:3037] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:30.812212 13079 raft_consensus.cc:2454] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a318f3b366674118a30bef8093f3bc90 in term 1.
I20260806 02:10:30.812212 12792 raft_consensus.cc:2454] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a318f3b366674118a30bef8093f3bc90 in term 1.
I20260806 02:10:30.812525 12863 leader_election.cc:302] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786, a318f3b366674118a30bef8093f3bc90; no voters:
I20260806 02:10:30.812737 13154 raft_consensus.cc:2781] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:30.812973 13154 raft_consensus.cc:686] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 LEADER]: Becoming Leader. State: Replica: a318f3b366674118a30bef8093f3bc90, State: Running, Role: LEADER
I20260806 02:10:30.813326 13154 consensus_queue.cc:237] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:30.816761 12635 catalog_manager.cc:5495] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 reported cstate change: term changed from 0 to 1, leader changed from <none> to a318f3b366674118a30bef8093f3bc90 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "a318f3b366674118a30bef8093f3bc90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } health_report { overall_health: UNKNOWN } } }
I20260806 02:10:30.840662 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.837133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.28.21.2:36395)
0806 02:10:30.837285 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:30.837288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:30.837361 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:10:30.837433 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:30.837435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:30.837498 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:30.837580 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:30.837791 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.838289 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.839224 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:30.839479 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:30.839561 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:30.839830 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:30.840230 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:30.840358 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:30.840384 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 02:10:30.840432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:10:30.846442 13079 raft_consensus.cc:1261] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Refusing update from remote peer a318f3b366674118a30bef8093f3bc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:30.846467 12792 raft_consensus.cc:1261] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Refusing update from remote peer a318f3b366674118a30bef8093f3bc90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:30.847173 13156 consensus_queue.cc:1035] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:30.847420 13154 consensus_queue.cc:1035] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:30.855278 13160 mvcc.cc:205] Tried to move back new op lower bound from 7315383217539837952 to 7315383217414111232. Current Snapshot: MvccSnapshot[applied={T|T < 7315383217539837952}]
I20260806 02:10:30.865564 12634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:30.865721 12634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:30.868209 12634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Sending DeleteTablet for 3 replicas of tablet 653080f0c8d34e2dbaed018b18a1e9c3
I20260806 02:10:30.868978 12916 tablet_service.cc:1514] Processing DeleteTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54506
I20260806 02:10:30.869099 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12705
I20260806 02:10:30.869084 12770 tablet_service.cc:1514] Processing DeleteTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:49282
I20260806 02:10:30.871045 13056 tablet_service.cc:1514] Processing DeleteTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:56054
I20260806 02:10:30.874670 13171 tablet_replica.cc:330] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: stopping tablet replica
I20260806 02:10:30.875063 13171 raft_consensus.cc:2229] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:30.875453 13171 raft_consensus.cc:2258] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:10:30.876940 12619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:42685 (error 108)
I20260806 02:10:30.877194 13171 ts_tablet_manager.cc:1905] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:30.877307 13170 tablet_replica.cc:330] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: stopping tablet replica
I20260806 02:10:30.877635 13170 raft_consensus.cc:2229] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:10:30.877666 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12845
I20260806 02:10:30.880293 13171 ts_tablet_manager.cc:1918] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:30.880415 13171 log.cc:1196] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/653080f0c8d34e2dbaed018b18a1e9c3
I20260806 02:10:30.880786 13171 ts_tablet_manager.cc:1939] T 653080f0c8d34e2dbaed018b18a1e9c3 P b6beb872c50643f4909d395beba78c4c: Deleting consensus metadata
I20260806 02:10:30.881332 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.880884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.0.0.1:49290)
0806 02:10:30.881075 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.881192 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 02:10:30.881554 12619 catalog_manager.cc:4584] TS 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 653080f0c8d34e2dbaed018b18a1e9c3: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42685: connect: Connection refused (error 111)
I20260806 02:10:30.882356 12621 catalog_manager.cc:4841] TS b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): tablet 653080f0c8d34e2dbaed018b18a1e9c3 (table test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]) successfully deleted
I20260806 02:10:30.887924 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12989
I20260806 02:10:30.888123 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:30.887834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36395 (local address 127.0.0.1:54520)
0806 02:10:30.887965 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:30.888021 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.2:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:10:30.895326 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 12603
I20260806 02:10:50.903403 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--webserver_interface=127.28.21.62
--webserver_port=45591
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:10:51.028093 13173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:51.028379 13173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:51.028424 13173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:51.032989 13173 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:10:51.033051 13173 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:10:51.033068 13173 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:10:51.033084 13173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:10:51.033105 13173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:51.033164 13173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:51.033196 13173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:10:51.033233 13173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:10:51.039175 13173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:51.040328 13173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:51.041458 13173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:51.055266 13181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:51.055339 13173 server_base.cc:1034] running on GCE node
W20260806 02:10:51.055421 13180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.055533 13183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:51.056107 13173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:51.057286 13173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:51.058478 13173 hybrid_clock.cc:648] HybridClock initialized: now 1785982251058463 us; error 29 us; skew 500 ppm
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:51.067492 13173 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:10:51.068536 13173 webserver.cc:460] Webserver started at http://127.28.21.62:45591/ using document root <none> and password file <none>
I20260806 02:10:51.069212 13173 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:51.069343 13173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:51.073536 13173 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:10:51.075835 13190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.077114 13173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:10:51.077260 13173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "e7431e4ed6554737b486ff8208ccc2f0"
format_stamp: "Formatted at 2026-08-06 02:10:29 on dist-test-slave-gp6n"
server_key: "64501afab4ce29ba8a1b45602556d243"
server_key_iv: "cb60aa982b24547f0cafad8484c6ae15"
server_key_version: "encryptionkey@0"
I20260806 02:10:51.077872 13173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:51.089913 13173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:51.093179 13173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:51.093390 13173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:51.100375 13173 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:44115
I20260806 02:10:51.100688 13244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:44115 every 8 connection(s)
I20260806 02:10:51.101552 13173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:10:51.106009 13245 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:10:51.107744 13245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap starting.
I20260806 02:10:51.111321 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13173
I20260806 02:10:51.111819 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:42685
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=44099
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:51.126386 13245 log.cc:824] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:51.135317 13245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:51.135725 13245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap complete.
I20260806 02:10:51.139958 13245 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:51.140342 13245 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.140817 13245 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:51.140929 13245 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:10:51.141031 13245 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:10:51.141144 13245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:10:51.143096 13245 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:51.143428 13245 leader_election.cc:302] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7431e4ed6554737b486ff8208ccc2f0; no voters:
I20260806 02:10:51.143771 13245 leader_election.cc:288] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:10:51.143852 13249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:10:51.144062 13249 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 LEADER]: Becoming Leader. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Running, Role: LEADER
I20260806 02:10:51.144349 13249 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:10:51.145023 13245 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:10:51.145283 13249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:10:51.145406 13249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:51.145772 13253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:10:51.145889 13250 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431e4ed6554737b486ff8208ccc2f0. Latest consensus state: current_term: 2 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:10:51.146071 13250 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:10:51.150717 13253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:10:51.151531 13253 catalog_manager.cc:1259] Loaded cluster ID: 6c89c2a8f8874f47a64b8bcf68d214b8
I20260806 02:10:51.151593 13253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:10:51.154062 13253 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:10:51.154793 13253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Loaded TSK: 0
I20260806 02:10:51.168975 13253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Generated new TSK 1
I20260806 02:10:51.169230 13253 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:10:51.230037 13247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:51.230396 13247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:51.230523 13247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:51.234891 13247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:51.234973 13247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:51.235061 13247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:10:51.240410 13247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:42685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=44099
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:51.241451 13247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:51.242517 13247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:51.253425 13275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.253489 13276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.253428 13278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:51.253976 13247 server_base.cc:1034] running on GCE node
I20260806 02:10:51.254364 13247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:51.255034 13247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:51.256217 13247 hybrid_clock.cc:648] HybridClock initialized: now 1785982251256188 us; error 43 us; skew 500 ppm
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:51.259248 13247 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:10:51.260201 13247 webserver.cc:460] Webserver started at http://127.28.21.1:44099/ using document root <none> and password file <none>
I20260806 02:10:51.260776 13247 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:51.260872 13247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:51.264669 13247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:10:51.266855 13285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.268095 13247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:10:51.268191 13247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "5f7eba7b375c463da0b4dbba11e4f786"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "3158a4eceeb96e3eac2c7fa001e7739c"
server_key_iv: "c0b35319f63cdd9fca377f54ac772c5c"
server_key_version: "encryptionkey@0"
I20260806 02:10:51.268757 13247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:51.283496 13247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:51.286633 13247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:51.286829 13247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:51.287469 13247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:51.288609 13292 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:10:51.292658 13247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:10:51.292718 13247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260806 02:10:51.292764 13247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:10:51.294348 13247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:10:51.294432 13247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:51.294687 13292 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap starting.
I20260806 02:10:51.305445 13247 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:42685
I20260806 02:10:51.305537 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:42685 every 8 connection(s)
I20260806 02:10:51.306752 13247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:10:51.309641 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13247
I20260806 02:10:51.310129 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:36395
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=40287
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:10:51.316126 13292 log.cc:824] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:51.323987 13292 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:51.324440 13292 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap complete.
I20260806 02:10:51.325984 13292 ts_tablet_manager.cc:1397] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.011s
I20260806 02:10:51.328786 13292 raft_consensus.cc:348] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:51.329334 13292 raft_consensus.cc:729] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7eba7b375c463da0b4dbba11e4f786, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.329569 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.308928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:50465)
0806 02:10:51.310716 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.310782 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.311891 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.313581 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.313593 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.314168 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:51.314847 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.314867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.317182 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.317186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.318547 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.318559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.319802 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.319805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.319989 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.320748 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:51.320767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:51.322997 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:51.325906 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.325913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.325928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.326338 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.326792 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.326796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.326798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.326985 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.327686 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:51.327693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:51.327996 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.328195 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.328549 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":170,"threads_started":1}
I20260806 02:10:51.329969 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.308931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:50465 (local address 127.28.21.62:44115)
0806 02:10:51.309359 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.309384 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.310954 (+ 1570us) server_negotiation.cc:407] Connection header received
0806 02:10:51.312440 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.312462 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.312926 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.313372 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.315063 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.316997 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.318692 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.319634 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.319854 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.323263 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.323270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.323286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.323320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.325770 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.326519 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.326525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.326531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.326682 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.327108 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.327112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.327113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.327562 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.327695 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.328361 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.328668 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":146,"threads_started":1}
I20260806 02:10:51.329892 13292 consensus_queue.cc:260] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:51.330943 13292 ts_tablet_manager.cc:1428] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260806 02:10:51.332381 13410 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:51.332628 13410 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:51.333084 13410 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
I20260806 02:10:51.335433 13207 ts_manager.cc:194] Registered new tserver with Master: 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685)
I20260806 02:10:51.338624 13207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:50465
I20260806 02:10:51.341126 13410 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.1@KRBTEST.COM
I20260806 02:10:51.349252 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.338184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.0.0.1:54072)
0806 02:10:51.338658 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.338705 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.338825 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.339605 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.339612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.340072 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:51.340207 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.340215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.341481 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.341484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.342172 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.342181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.342920 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.342924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.343045 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.343604 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:51.343620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:51.345465 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:51.347523 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.347534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.347539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.347765 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.348074 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.348078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.348080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.348169 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.348621 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:51.348626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:51.348742 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.348877 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.349068 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":159,"threads_started":1}
I20260806 02:10:51.349264 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.338258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.28.21.1:42685)
0806 02:10:51.339096 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.339107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.339127 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:10:51.339188 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.339192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.339386 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.339545 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.340346 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.341341 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.342294 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.342813 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.342914 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.345603 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.345611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.345620 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.345754 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.347397 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.347881 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.347887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.347889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.347979 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.348260 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.348265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.348266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.348498 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.348631 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.348967 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.349118 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":103,"threads_started":1}
I20260806 02:10:51.350203 13335 tablet_service.cc:1514] Processing DeleteTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54072
I20260806 02:10:51.351074 13420 tablet_replica.cc:330] stopping tablet replica
I20260806 02:10:51.351365 13420 raft_consensus.cc:2229] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:51.351609 13420 raft_consensus.cc:2258] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:51.352532 13420 ts_tablet_manager.cc:1905] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:51.356161 13420 ts_tablet_manager.cc:1918] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:51.356254 13420 log.cc:1196] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal/wals/653080f0c8d34e2dbaed018b18a1e9c3
I20260806 02:10:51.356613 13420 ts_tablet_manager.cc:1939] T 653080f0c8d34e2dbaed018b18a1e9c3 P 5f7eba7b375c463da0b4dbba11e4f786: Deleting consensus metadata
I20260806 02:10:51.358609 13191 catalog_manager.cc:4841] TS 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685): tablet 653080f0c8d34e2dbaed018b18a1e9c3 (table test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]) successfully deleted
W20260806 02:10:51.442512 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:51.442860 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:51.442948 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:51.447933 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:51.448033 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:51.448135 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:10:51.453711 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:36395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=40287
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:51.454846 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:51.455904 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:51.465713 13425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.465698 13426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.465716 13428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:51.465884 13414 server_base.cc:1034] running on GCE node
I20260806 02:10:51.466349 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:51.467087 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:51.468256 13414 hybrid_clock.cc:648] HybridClock initialized: now 1785982251468223 us; error 45 us; skew 500 ppm
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:51.471297 13414 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:10:51.472415 13414 webserver.cc:460] Webserver started at http://127.28.21.2:40287/ using document root <none> and password file <none>
I20260806 02:10:51.473063 13414 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:51.473176 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:51.477025 13414 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 02:10:51.479254 13436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.480418 13414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:51.480571 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a318f3b366674118a30bef8093f3bc90"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "1033c164230ac8e92e7e2022522d306b"
server_key_iv: "25e917ba5efc16183154fd35cd2b1923"
server_key_version: "encryptionkey@0"
I20260806 02:10:51.481118 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:51.511585 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:51.515009 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:51.515214 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:51.515815 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:51.516932 13443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:10:51.521466 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:10:51.521529 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 02:10:51.521626 13414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:10:51.523377 13414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:10:51.523422 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260806 02:10:51.523743 13443 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Bootstrap starting.
I20260806 02:10:51.533571 13414 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:36395
I20260806 02:10:51.533620 13556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:36395 every 8 connection(s)
I20260806 02:10:51.534530 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:10:51.538960 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13414
I20260806 02:10:51.539391 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:44907
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=43187
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:51.546765 13443 log.cc:824] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Log is configured to *not* fsync() on all Append() calls
I20260806 02:10:51.549690 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.536360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:41827 (local address 127.28.21.62:44115)
0806 02:10:51.536538 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.536544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.537242 (+ 698us) server_negotiation.cc:407] Connection header received
0806 02:10:51.538527 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.538530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.538579 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.538644 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.540424 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.541273 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.542449 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.542701 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.542825 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.545347 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.545351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.545353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.545381 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.547150 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.547675 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.547678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.547680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.547769 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.548174 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.548182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.548186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.548504 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.548641 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.549225 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.549429 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:10:51.550787 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.536441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:41827)
0806 02:10:51.537035 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.537106 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.538270 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.538888 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.538901 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.539471 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:51.540216 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.540247 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.541433 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.541437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.542291 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.542300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.542841 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.542845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.543029 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.543641 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:51.543657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:51.545171 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:51.547269 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.547277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.547291 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.547567 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.547860 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.547864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.547866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.548042 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.548640 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:51.548646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:51.548930 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.549129 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.549487 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260806 02:10:51.552053 13557 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:51.552271 13557 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:51.552855 13557 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
I20260806 02:10:51.554713 13207 ts_manager.cc:194] Registered new tserver with Master: a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
I20260806 02:10:51.555518 13443 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:10:51.555902 13443 tablet_bootstrap.cc:492] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Bootstrap complete.
I20260806 02:10:51.556043 13207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:41827
I20260806 02:10:51.557289 13443 ts_tablet_manager.cc:1397] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260806 02:10:51.559949 13443 raft_consensus.cc:348] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:51.560354 13443 raft_consensus.cc:729] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a318f3b366674118a30bef8093f3bc90, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.560810 13443 consensus_queue.cc:260] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } } peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } }
I20260806 02:10:51.561657 13557 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
I20260806 02:10:51.561873 13443 ts_tablet_manager.cc:1428] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.2@KRBTEST.COM
I20260806 02:10:51.567544 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.555714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36395 (local address 127.0.0.1:56882)
0806 02:10:51.555995 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.556010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.556091 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.557067 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.557073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.557274 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:51.557386 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.557394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.559202 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.559206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.560258 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.560269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.561182 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.561186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.561280 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.561847 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:51.561862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:51.563574 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:51.565895 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.565897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.565899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.566092 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.566475 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.566478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.566480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.566567 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.567076 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:51.567078 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:51.567156 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.567238 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.567387 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260806 02:10:51.567706 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.556246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.28.21.2:36395)
0806 02:10:51.556601 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.556615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.556639 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:10:51.556706 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.556714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.556908 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.557553 (+ 645us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.557975 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.559070 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.560418 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.561027 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.561163 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.563736 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.563746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.563760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.563786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.565775 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.566210 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.566216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.566219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.566328 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.566659 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.566663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.566666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.566953 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.567097 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.567361 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.567515 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260806 02:10:51.568774 13488 tablet_service.cc:1514] Processing DeleteTablet for tablet 653080f0c8d34e2dbaed018b18a1e9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:56882
I20260806 02:10:51.569561 13565 tablet_replica.cc:330] stopping tablet replica
I20260806 02:10:51.569844 13565 raft_consensus.cc:2229] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:51.570075 13565 raft_consensus.cc:2258] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:51.571053 13565 ts_tablet_manager.cc:1905] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:51.574677 13565 ts_tablet_manager.cc:1918] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:51.574760 13565 log.cc:1196] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/653080f0c8d34e2dbaed018b18a1e9c3
I20260806 02:10:51.575032 13565 ts_tablet_manager.cc:1939] T 653080f0c8d34e2dbaed018b18a1e9c3 P a318f3b366674118a30bef8093f3bc90: Deleting consensus metadata
I20260806 02:10:51.576583 13194 catalog_manager.cc:4841] TS a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): tablet 653080f0c8d34e2dbaed018b18a1e9c3 (table test-table [id=bbc128dfc5db4f70813ad1ed194d1c44]) successfully deleted
W20260806 02:10:51.662621 13561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:10:51.662894 13561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:10:51.662968 13561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:10:51.666895 13561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:10:51.666983 13561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:10:51.667070 13561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:10:51.672699 13561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:44907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=43187
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:10:51.673794 13561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:10:51.674871 13561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:10:51.684346 13574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.684404 13571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:10:51.684516 13570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:10:51.684767 13561 server_base.cc:1034] running on GCE node
I20260806 02:10:51.685237 13561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:10:51.685985 13561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:10:51.687184 13561 hybrid_clock.cc:648] HybridClock initialized: now 1785982251687156 us; error 47 us; skew 500 ppm
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:10:51.689925 13561 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:10:51.690843 13561 webserver.cc:460] Webserver started at http://127.28.21.3:43187/ using document root <none> and password file <none>
I20260806 02:10:51.691336 13561 fs_manager.cc:359] Metadata directory not provided
I20260806 02:10:51.691402 13561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:10:51.695178 13561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:10:51.697080 13582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.698226 13561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:10:51.698346 13561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "b6beb872c50643f4909d395beba78c4c"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "5f992fa0723d3874628df9e8349493bb"
server_key_iv: "b0ab453a665a22097abde9c5362fc7fc"
server_key_version: "encryptionkey@0"
I20260806 02:10:51.698927 13561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:10:51.722270 13561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:10:51.725203 13561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:10:51.725412 13561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:10:51.726017 13561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:10:51.726850 13561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:10:51.726897 13561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.726979 13561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:10:51.727022 13561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:10:51.736040 13561 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:44907
I20260806 02:10:51.736070 13705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:44907 every 8 connection(s)
I20260806 02:10:51.737047 13561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:10:51.737807 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13561
I20260806 02:10:51.747372 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.738647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.28.21.62:44115)
0806 02:10:51.738809 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.738815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.739072 (+ 257us) server_negotiation.cc:407] Connection header received
0806 02:10:51.739235 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.739239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.739294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.739382 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.740108 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.740706 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.741635 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.741974 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.742106 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.743743 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.743746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.743747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.743769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.745417 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.745944 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.745946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.745948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.746058 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.746360 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.746363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.746364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.746646 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.746852 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.747071 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.747200 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 02:10:51 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982251, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:10:51.752336 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.739081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:58283 (local address 127.28.21.62:44115)
0806 02:10:51.739319 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.739322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.740234 (+ 912us) server_negotiation.cc:407] Connection header received
0806 02:10:51.741540 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.741546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.741608 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.741700 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.743053 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.743888 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.745690 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.746058 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.746165 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.748876 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.748880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.748882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.748906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.750273 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.750809 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.750813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.750814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.750904 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.751256 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.751259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.751261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.751435 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.751551 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.752046 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.752166 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260806 02:10:51.753142 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.739617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:58283)
0806 02:10:51.740084 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.740134 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.741332 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.741892 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.741901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.742374 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:10:51.742895 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.742909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.744035 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.744037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.745572 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.745581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.746216 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.746224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.746466 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.747052 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:10:51.747072 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:10:51.748714 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:10:51.750394 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.750416 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.750431 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.750700 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.751003 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:10:51.751007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:10:51.751009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:10:51.751151 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:10:51.751550 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:10:51.751557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:10:51.751803 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.751993 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.752310 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":139,"threads_started":1}
I20260806 02:10:51.754165 13706 heartbeater.cc:347] Connected to a master server at 127.28.21.62:44115
I20260806 02:10:51.754467 13706 heartbeater.cc:464] Registering TS with master...
I20260806 02:10:51.754952 13706 heartbeater.cc:511] Master 127.28.21.62:44115 requested a full tablet report, sending...
I20260806 02:10:51.755908 13207 ts_manager.cc:194] Registered new tserver with Master: b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907)
I20260806 02:10:51.756446 13207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:58283
I20260806 02:10:51.760851 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:10:51.764699 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.761446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.28.21.62:44115)
0806 02:10:51.761604 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.761611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.761767 (+ 156us) server_negotiation.cc:407] Connection header received
0806 02:10:51.761891 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.761893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.761980 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.762062 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:51.762306 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.762861 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.763698 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.763910 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.763985 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.764231 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:51.764349 (+ 118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:10:51.764549 (+ 200us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40294: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:10:51.764780 13709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40294: token expired
W20260806 02:10:51.765105 13132 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.21.62:44115)
I20260806 02:10:51.772643 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.765453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.28.21.62:44115)
0806 02:10:51.765599 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.765603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.765616 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:10:51.765695 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.765698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.765737 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.765828 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:10:51.766480 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.766957 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.767734 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.767955 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.768056 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.769477 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:10:51.769481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:10:51.769483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:10:51.769513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:10:51.770935 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.771350 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.771354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.771355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.771438 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:10:51.771719 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:10:51.771723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:10:51.771724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:10:51.771943 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:10:51.772090 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.772331 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.772472 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:10:51.775141 13207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:10:51.778360 13207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:10:51.786284 13488 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda (DEFAULT_TABLE table=test-table [id=ff4be91566104a5cba37857e199e649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:51.787024 13488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:51.789299 13443 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Bootstrap starting.
I20260806 02:10:51.789758 13335 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda (DEFAULT_TABLE table=test-table [id=ff4be91566104a5cba37857e199e649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:51.790187 13443 tablet_bootstrap.cc:654] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:51.790369 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:51.790654 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.785553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:44907 (local address 127.0.0.1:40844)
0806 02:10:51.785751 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.785767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.785843 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.786993 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.786996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.787017 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:51.787127 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.787135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.788216 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.788223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.789016 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.789025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.790147 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.790150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.790248 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.790302 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.790391 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.790533 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 02:10:51.790702 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.785901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.28.21.3:44907)
0806 02:10:51.786245 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.786259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.786294 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:10:51.786359 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.786364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.786655 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.787162 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:51.787354 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.788097 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.789154 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.789994 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.790139 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.790324 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.790440 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.790541 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":159,"threads_started":1}
I20260806 02:10:51.792068 13443 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: No bootstrap required, opened a new log
I20260806 02:10:51.792174 13443 ts_tablet_manager.cc:1397] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:10:51.792615 13443 raft_consensus.cc:348] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.792759 13443 raft_consensus.cc:374] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:51.792821 13443 raft_consensus.cc:729] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a318f3b366674118a30bef8093f3bc90, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.792984 13443 consensus_queue.cc:260] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.793107 13292 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap starting.
I20260806 02:10:51.793293 13443 ts_tablet_manager.cc:1428] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:10:51.794015 13292 tablet_bootstrap.cc:654] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:51.794395 13631 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda (DEFAULT_TABLE table=test-table [id=ff4be91566104a5cba37857e199e649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:10:51.795542 13292 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: No bootstrap required, opened a new log
I20260806 02:10:51.795552 13631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:10:51.795639 13292 ts_tablet_manager.cc:1397] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:10:51.796134 13292 raft_consensus.cc:348] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.796260 13292 raft_consensus.cc:374] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:51.796308 13292 raft_consensus.cc:729] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7eba7b375c463da0b4dbba11e4f786, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.796446 13292 consensus_queue.cc:260] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.796725 13292 ts_tablet_manager.cc:1428] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:10:51.803049 13721 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Bootstrap starting.
I20260806 02:10:51.805212 13721 tablet_bootstrap.cc:654] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Neither blocks nor log segments found. Creating new log.
I20260806 02:10:51.806272 13721 log.cc:824] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Log is configured to *not* fsync() on all Append() calls
W20260806 02:10:51.807934 13411 tablet.cc:2369] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:51.808207 13721 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: No bootstrap required, opened a new log
I20260806 02:10:51.808436 13721 ts_tablet_manager.cc:1397] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:10:51.811479 13721 raft_consensus.cc:348] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.811755 13721 raft_consensus.cc:374] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:10:51.811839 13721 raft_consensus.cc:729] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6beb872c50643f4909d395beba78c4c, State: Initialized, Role: FOLLOWER
I20260806 02:10:51.812278 13721 consensus_queue.cc:260] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.813030 13706 heartbeater.cc:503] Master 127.28.21.62:44115 was elected leader, sending a full tablet report...
I20260806 02:10:51.813316 13721 ts_tablet_manager.cc:1428] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:10:51.981866 13723 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:10:51.982102 13723 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.983335 13723 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
W20260806 02:10:51.987912 13707 tablet.cc:2369] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:51.988405 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.983577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.28.21.3:59627)
0806 02:10:51.983692 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.983710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.983802 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.984207 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.984213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.984232 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:51.984364 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.984370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.985518 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.985525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.987049 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.987060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.988002 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.988005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.988110 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.988134 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.988210 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.988279 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":135}
I20260806 02:10:51.988657 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.983647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:59627 (local address 127.28.21.1:42685)
0806 02:10:51.983931 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.983934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.983946 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:10:51.983997 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.983999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.984052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.984177 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:51.984507 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.985373 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.987181 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.987866 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.987994 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.988138 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.988326 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.988411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
I20260806 02:10:51.988688 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.983692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:48985 (local address 127.28.21.2:36395)
0806 02:10:51.983838 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:51.983844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:51.984135 (+ 291us) server_negotiation.cc:407] Connection header received
0806 02:10:51.984277 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:51.984281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:51.984345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:51.984418 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:10:51.984706 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.985443 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.987181 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.987871 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.987980 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.988137 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:51.988492 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:10:51.988580 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:10:51.989045 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:51.983587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36395 (local address 127.28.21.3:48985)
0806 02:10:51.984027 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:51.984038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:10:51.984135 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:10:51.984457 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:10:51.984460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:10:51.984510 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:10:51.984596 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.984600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.985564 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.985567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.987049 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:10:51.987060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:10:51.988226 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:10:51.988230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:10:51.988345 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:51.988367 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:10:51.988799 (+ 432us) client_negotiation.cc:240] Negotiation successful
0806 02:10:51.988902 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":181,"thread_start_us":116,"threads_started":1}
I20260806 02:10:51.989246 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab5bb5bdd9e4336a4014b36e8cceeda" candidate_uuid: "b6beb872c50643f4909d395beba78c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7eba7b375c463da0b4dbba11e4f786" is_pre_election: true
I20260806 02:10:51.989616 13508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab5bb5bdd9e4336a4014b36e8cceeda" candidate_uuid: "b6beb872c50643f4909d395beba78c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a318f3b366674118a30bef8093f3bc90" is_pre_election: true
I20260806 02:10:51.989677 13356 raft_consensus.cc:2454] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6beb872c50643f4909d395beba78c4c in term 0.
I20260806 02:10:51.989883 13508 raft_consensus.cc:2454] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6beb872c50643f4909d395beba78c4c in term 0.
I20260806 02:10:51.990222 13583 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786, b6beb872c50643f4909d395beba78c4c; no voters:
I20260806 02:10:51.990581 13723 raft_consensus.cc:2781] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:10:51.990679 13723 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:10:51.990727 13723 raft_consensus.cc:3037] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:51.991825 13723 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.992199 13723 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [CANDIDATE]: Term 1 election: Requested vote from peers 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
I20260806 02:10:51.992496 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab5bb5bdd9e4336a4014b36e8cceeda" candidate_uuid: "b6beb872c50643f4909d395beba78c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7eba7b375c463da0b4dbba11e4f786"
I20260806 02:10:51.992580 13508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab5bb5bdd9e4336a4014b36e8cceeda" candidate_uuid: "b6beb872c50643f4909d395beba78c4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a318f3b366674118a30bef8093f3bc90"
I20260806 02:10:51.992633 13356 raft_consensus.cc:3037] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:51.992743 13508 raft_consensus.cc:3037] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:10:51.993777 13508 raft_consensus.cc:2454] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6beb872c50643f4909d395beba78c4c in term 1.
I20260806 02:10:51.993777 13356 raft_consensus.cc:2454] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6beb872c50643f4909d395beba78c4c in term 1.
I20260806 02:10:51.994083 13586 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c; no voters:
I20260806 02:10:51.994247 13723 raft_consensus.cc:2781] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:10:51.994519 13723 raft_consensus.cc:686] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 1 LEADER]: Becoming Leader. State: Replica: b6beb872c50643f4909d395beba78c4c, State: Running, Role: LEADER
I20260806 02:10:51.994840 13723 consensus_queue.cc:237] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:10:51.998114 13207 catalog_manager.cc:5495] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c reported cstate change: term changed from 0 to 1, leader changed from <none> to b6beb872c50643f4909d395beba78c4c (127.28.21.3). New cstate: current_term: 1 leader_uuid: "b6beb872c50643f4909d395beba78c4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } health_report { overall_health: UNKNOWN } } }
I20260806 02:10:52.033196 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:52.029704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.28.21.3:44907)
0806 02:10:52.029839 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:10:52.029842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:10:52.029857 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:10:52.029932 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:10:52.029934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:10:52.029989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:10:52.030107 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:10:52.030359 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:52.030886 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:52.031721 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:10:52.031943 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:10:52.032052 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:10:52.032358 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:10:52.032784 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:10:52.032923 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:10:52.032964 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 02:10:52.033024 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 02:10:52.035688 13559 tablet.cc:2369] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:10:52.039319 13508 raft_consensus.cc:1261] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Refusing update from remote peer b6beb872c50643f4909d395beba78c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:52.039321 13356 raft_consensus.cc:1261] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Refusing update from remote peer b6beb872c50643f4909d395beba78c4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:10:52.040025 13723 consensus_queue.cc:1035] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:52.040242 13726 consensus_queue.cc:1035] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:10:52.055637 13207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:52.055820 13207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:10:52.058220 13207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Sending DeleteTablet for 3 replicas of tablet 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:10:52.059059 13631 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:52 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:40844
I20260806 02:10:52.059110 13335 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:52 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:54072
I20260806 02:10:52.059505 13739 tablet_replica.cc:330] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: stopping tablet replica
I20260806 02:10:52.059499 13488 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:10:52 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:56882
I20260806 02:10:52.059676 13565 tablet_replica.cc:330] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: stopping tablet replica
I20260806 02:10:52.059712 13739 raft_consensus.cc:2229] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:52.059803 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 13247
I20260806 02:10:52.059804 13565 raft_consensus.cc:2229] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:10:52.059943 13565 raft_consensus.cc:2258] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:52.060070 13739 raft_consensus.cc:2258] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:52.061126 13565 ts_tablet_manager.cc:1905] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:52.063815 13565 ts_tablet_manager.cc:1918] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:52.063901 13565 log.cc:1196] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:10:52.064186 13565 ts_tablet_manager.cc:1939] T 3ab5bb5bdd9e4336a4014b36e8cceeda P a318f3b366674118a30bef8093f3bc90: Deleting consensus metadata
I20260806 02:10:52.064496 13740 tablet_replica.cc:330] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: stopping tablet replica
I20260806 02:10:52.064812 13740 raft_consensus.cc:2229] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:10:52.065102 13194 catalog_manager.cc:4841] TS a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): tablet 3ab5bb5bdd9e4336a4014b36e8cceeda (table test-table [id=ff4be91566104a5cba37857e199e649c]) successfully deleted
I20260806 02:10:52.065173 13740 raft_consensus.cc:2258] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:10:52.066568 13740 ts_tablet_manager.cc:1905] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:10:52.069136 13740 ts_tablet_manager.cc:1918] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:10:52.069234 13740 log.cc:1196] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:10:52.069720 13191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:42685 (error 108)
I20260806 02:10:52.070066 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 13414
I20260806 02:10:52.070241 13740 ts_tablet_manager.cc:1939] T 3ab5bb5bdd9e4336a4014b36e8cceeda P b6beb872c50643f4909d395beba78c4c: Deleting consensus metadata
I20260806 02:10:52.071024 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:10:52.070511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:42685 (local address 127.0.0.1:51220)
0806 02:10:52.070837 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 02:10:52.070902 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
W20260806 02:10:52.071233 13191 catalog_manager.cc:4584] TS 5f7eba7b375c463da0b4dbba11e4f786 (127.28.21.1:42685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ab5bb5bdd9e4336a4014b36e8cceeda: Network error: Client connection negotiation failed: client connection to 127.28.21.1:42685: connect: Connection refused (error 111)
I20260806 02:10:52.072991 13193 catalog_manager.cc:4841] TS b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): tablet 3ab5bb5bdd9e4336a4014b36e8cceeda (table test-table [id=ff4be91566104a5cba37857e199e649c]) successfully deleted
I20260806 02:10:52.077311 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 13561
I20260806 02:10:52.085906 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 13173
I20260806 02:11:12.094105 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--webserver_interface=127.28.21.62
--webserver_port=45591
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:12.206693 13742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:12.207021 13742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:12.207106 13742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:12.211133 13742 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:11:12.211212 13742 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:11:12.211243 13742 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:11:12.211277 13742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:12.211315 13742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:12.211351 13742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:12.211385 13742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:12.211416 13742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:12.218101 13742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:12.219704 13742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:12.221064 13742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:12.231007 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.231037 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.231063 13742 server_base.cc:1034] running on GCE node
W20260806 02:11:12.231305 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.231873 13742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:12.232987 13742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:12.234153 13742 hybrid_clock.cc:648] HybridClock initialized: now 1785982272234137 us; error 30 us; skew 500 ppm
Aug 06 02:11:12 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:12.243165 13742 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:12.244210 13742 webserver.cc:460] Webserver started at http://127.28.21.62:45591/ using document root <none> and password file <none>
I20260806 02:11:12.244813 13742 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:12.244880 13742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:12.248874 13742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:11:12.251157 13757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.252462 13742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260806 02:11:12.252604 13742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "e7431e4ed6554737b486ff8208ccc2f0"
format_stamp: "Formatted at 2026-08-06 02:10:29 on dist-test-slave-gp6n"
server_key: "64501afab4ce29ba8a1b45602556d243"
server_key_iv: "cb60aa982b24547f0cafad8484c6ae15"
server_key_version: "encryptionkey@0"
I20260806 02:11:12.253162 13742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:12.265623 13742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:12.268829 13742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:12.269040 13742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:12.276393 13742 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:44115
I20260806 02:11:12.276701 13818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:44115 every 8 connection(s)
I20260806 02:11:12.277493 13742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:12.281889 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13742
I20260806 02:11:12.282395 13819 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:11:12.282454 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:42685
--local_ip_for_outbound_sockets=127.28.21.1
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=44099
--webserver_interface=127.28.21.1
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:11:12.284880 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap starting.
I20260806 02:11:12.305019 13819 log.cc:824] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:12.316552 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:11:12.316938 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Bootstrap complete.
I20260806 02:11:12.321102 13819 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:11:12.321508 13819 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Initialized, Role: FOLLOWER
I20260806 02:11:12.321993 13819 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:11:12.322142 13819 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:12.322228 13819 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:12.322350 13819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:11:12.324232 13819 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:11:12.324568 13819 leader_election.cc:302] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7431e4ed6554737b486ff8208ccc2f0; no voters:
I20260806 02:11:12.324918 13819 leader_election.cc:288] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:11:12.325003 13823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:11:12.325194 13823 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [term 3 LEADER]: Becoming Leader. State: Replica: e7431e4ed6554737b486ff8208ccc2f0, State: Running, Role: LEADER
I20260806 02:11:12.325490 13823 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } }
I20260806 02:11:12.326149 13819 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:12.326419 13823 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431e4ed6554737b486ff8208ccc2f0. Latest consensus state: current_term: 3 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:11:12.326534 13823 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:12.327005 13824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7431e4ed6554737b486ff8208ccc2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7431e4ed6554737b486ff8208ccc2f0" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44115 } } }
I20260806 02:11:12.327193 13824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:12.328341 13827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:12.332705 13827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:12.333240 13827 catalog_manager.cc:1259] Loaded cluster ID: 6c89c2a8f8874f47a64b8bcf68d214b8
I20260806 02:11:12.333325 13827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:12.335152 13827 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:12.335777 13827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Loaded TSK: 0
I20260806 02:11:12.335865 13827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7431e4ed6554737b486ff8208ccc2f0: Loaded TSK: 1
F20260806 02:11:12.336454 13827 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc54998d9ad google::LogMessage::Fail() at ??:0
@ 0x7fc54999088a google::LogMessage::SendToLog() at ??:0
@ 0x7fc54998d4de google::LogMessage::Flush() at ??:0
@ 0x7fc54998ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc54c65118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc54c649f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc5542f27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc5542cc99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc5542cd178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc55430edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc55432d82b std::function<>::operator()() at ??:0
@ 0x7fc5542ccb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc5542cdf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc5542ca579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc55430e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc54bbdf3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc54bbdfc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc54bbe13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc54bbd0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc54d2146db start_thread at ??:0
@ 0x7fc54848b61f clone at ??:0
W20260806 02:11:12.431577 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:12.431886 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:12.431932 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:12.436251 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:12.436311 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:12.436371 13821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:12.441938 13821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:42685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=44099
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:12.443159 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:12.444247 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:12.454206 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.454206 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.454574 13821 server_base.cc:1034] running on GCE node
W20260806 02:11:12.454679 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.455155 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:12.456043 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:12.457266 13821 hybrid_clock.cc:648] HybridClock initialized: now 1785982272457228 us; error 46 us; skew 500 ppm
Aug 06 02:11:12 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:12.461001 13821 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:11:12.461942 13821 webserver.cc:460] Webserver started at http://127.28.21.1:44099/ using document root <none> and password file <none>
I20260806 02:11:12.462598 13821 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:12.462666 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:12.466727 13821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:12.469096 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.470626 13821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:12.470767 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "5f7eba7b375c463da0b4dbba11e4f786"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "3158a4eceeb96e3eac2c7fa001e7739c"
server_key_iv: "c0b35319f63cdd9fca377f54ac772c5c"
server_key_version: "encryptionkey@0"
I20260806 02:11:12.471333 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:12.500272 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:12.503882 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:12.504105 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:12.504738 13821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:12.505890 13883 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:11:12.511780 13821 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:11:12.511857 13821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 02:11:12.511960 13821 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:11:12.513837 13821 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:11:12.513912 13821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:11:12.514264 13883 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap starting.
I20260806 02:11:12.524698 13821 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:42685
I20260806 02:11:12.525650 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:12.526717 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:42685 every 8 connection(s)
I20260806 02:11:12.531654 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 13821
I20260806 02:11:12.532109 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:36395
--local_ip_for_outbound_sockets=127.28.21.2
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=40287
--webserver_interface=127.28.21.2
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:11:12.540838 13883 log.cc:824] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:12.548323 13883 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:11:12.548720 13883 tablet_bootstrap.cc:492] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Bootstrap complete.
I20260806 02:11:12.549889 13883 ts_tablet_manager.cc:1397] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.017s
I20260806 02:11:12.552486 13883 raft_consensus.cc:348] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:12.552927 13883 raft_consensus.cc:729] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7eba7b375c463da0b4dbba11e4f786, State: Initialized, Role: FOLLOWER
I20260806 02:11:12.553435 13883 consensus_queue.cc:260] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:12.554486 13883 ts_tablet_manager.cc:1428] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786: Time spent starting tablet: real 0.004s user 0.007s sys 0.001s
W20260806 02:11:12.613183 14007 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:11:12.527767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:33871)
0806 02:11:12.528263 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.528318 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:12.529155 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:12.612206 (+ 83051us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":163,"threads_started":1}
W20260806 02:11:12.613858 14005 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:44115 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:44115: Client connection negotiation failed: client connection to 127.28.21.62:44115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:11:12.614676 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.614420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:60597)
0806 02:11:12.614553 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.614620 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:11:12.615375 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.615121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:53313)
0806 02:11:12.615244 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.615287 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:11:12.615593 14005 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:11:12.671339 14008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:12.671615 14008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:12.671659 14008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:12.675781 14008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:12.675859 14008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:12.675951 14008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:12.681203 14008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:36395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=40287
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:12.682282 14008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:12.683329 14008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:12.693017 14018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.693025 14020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.693042 14017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.693678 14008 server_base.cc:1034] running on GCE node
I20260806 02:11:12.694051 14008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:12.694720 14008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:12.695902 14008 hybrid_clock.cc:648] HybridClock initialized: now 1785982272695883 us; error 34 us; skew 500 ppm
Aug 06 02:11:12 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:12.698896 14008 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:11:12.699801 14008 webserver.cc:460] Webserver started at http://127.28.21.2:40287/ using document root <none> and password file <none>
I20260806 02:11:12.700382 14008 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:12.700448 14008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:12.704408 14008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:12.706614 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.707836 14008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:12.707959 14008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a318f3b366674118a30bef8093f3bc90"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "1033c164230ac8e92e7e2022522d306b"
server_key_iv: "25e917ba5efc16183154fd35cd2b1923"
server_key_version: "encryptionkey@0"
I20260806 02:11:12.708539 14008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:12.741178 14008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:12.744858 14008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:12.745095 14008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:12.745760 14008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:12.746697 14008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:12.746747 14008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.746816 14008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:12.746850 14008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.756240 14008 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:36395
I20260806 02:11:12.756320 14154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:36395 every 8 connection(s)
I20260806 02:11:12.757381 14008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:12.761793 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.759781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:36515)
0806 02:11:12.760335 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.760457 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
W20260806 02:11:12.762327 14155 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:44115 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:44115: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
I20260806 02:11:12.763023 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.762796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:35325)
0806 02:11:12.762913 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.762965 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 02:11:12.763787 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.763576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:42317)
0806 02:11:12.763688 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.763735 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:11:12.764019 14155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:11:12.764840 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14008
I20260806 02:11:12.765280 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:44907
--local_ip_for_outbound_sockets=127.28.21.3
--tserver_master_addrs=127.28.21.62:44115
--webserver_port=43187
--webserver_interface=127.28.21.3
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:12.882558 14158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:12.882844 14158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:12.882894 14158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:12.886968 14158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:12.887034 14158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:12.887094 14158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:12.892467 14158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:44907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=43187
--tserver_master_addrs=127.28.21.62:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:12.893466 14158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:12.894500 14158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:12.904377 14164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.904377 14163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:12.904444 14166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:12.904762 14158 server_base.cc:1034] running on GCE node
I20260806 02:11:12.905202 14158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:12.905929 14158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:12.907143 14158 hybrid_clock.cc:648] HybridClock initialized: now 1785982272907107 us; error 55 us; skew 500 ppm
Aug 06 02:11:12 dist-test-slave-gp6n krb5kdc[12590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:12.910027 14158 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:11:12.911067 14158 webserver.cc:460] Webserver started at http://127.28.21.3:43187/ using document root <none> and password file <none>
I20260806 02:11:12.911684 14158 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:12.911803 14158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:12.916004 14158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:12.918179 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.919467 14158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:12.919607 14158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "b6beb872c50643f4909d395beba78c4c"
format_stamp: "Formatted at 2026-08-06 02:10:30 on dist-test-slave-gp6n"
server_key: "5f992fa0723d3874628df9e8349493bb"
server_key_iv: "b0ab453a665a22097abde9c5362fc7fc"
server_key_version: "encryptionkey@0"
I20260806 02:11:12.920166 14158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:12.938499 14158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:12.941633 14158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:12.941855 14158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:12.942552 14158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:12.943400 14158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:12.943471 14158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.943542 14158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:12.943590 14158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:12.952771 14158 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:44907
I20260806 02:11:12.952796 14293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:44907 every 8 connection(s)
I20260806 02:11:12.953807 14158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:12.957780 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.955994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:54887)
0806 02:11:12.956492 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.956605 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
W20260806 02:11:12.958249 14294 heartbeater.cc:644] Failed to heartbeat to 127.28.21.62:44115 (0 consecutive failures): Network error: Failed to ping master at 127.28.21.62:44115: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
I20260806 02:11:12.958920 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.958684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:51609)
0806 02:11:12.958798 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.958870 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:11:12.959513 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:12.959315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:55493)
0806 02:11:12.959415 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:12.959460 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:11:12.959674 14294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:11:12.963392 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14158
I20260806 02:11:13.616737 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.616233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:56089)
0806 02:11:13.616563 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:13.616626 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260806 02:11:13.687965 14300 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:13.688215 14300 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:13.689386 14300 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
Aug 06 02:11:13 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.2@KRBTEST.COM
Aug 06 02:11:13 dist-test-slave-gp6n krb5kdc[12590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.3@KRBTEST.COM
I20260806 02:11:13.703195 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.689681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36395 (local address 127.28.21.1:58453)
0806 02:11:13.690160 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:13.690175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:13.690269 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:13.691701 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:13.691707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:13.692244 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:13.692703 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:13.692717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.694205 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.694208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:13.695246 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:13.695256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.695959 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.695963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:13.696197 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:13.696837 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:13.696853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:13.698460 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:13.700704 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:13.700710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:13.700722 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:13.700996 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:13.701390 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:13.701395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:13.701545 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:13.702151 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:13.702157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:13.702397 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 02:11:13.702650 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 02:11:13.703011 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":245,"spinlock_wait_cycles":14080,"thread_start_us":59,"threads_started":1}
I20260806 02:11:13.703208 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.689813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:58453 (local address 127.28.21.2:36395)
0806 02:11:13.690276 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:13.690287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:13.690362 (+ 75us) server_negotiation.cc:407] Connection header received
0806 02:11:13.691063 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:13.691069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:13.691364 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:13.691659 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:13.692869 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.694012 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.695366 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.695827 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.696020 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:13.698688 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:13.698693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:13.698704 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:13.698732 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:13.700577 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701116 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:13.701127 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:13.701150 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:13.701298 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701648 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:13.701651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:13.701653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:13.702030 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:13.702173 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:13.702763 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 02:11:13.703024 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":110,"threads_started":1}
I20260806 02:11:13.703231 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.689762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:49959 (local address 127.28.21.3:44907)
0806 02:11:13.690123 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:13.690139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:13.690188 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:11:13.690972 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:13.690978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:13.691326 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:13.691621 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:13.692868 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.694026 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.695382 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.695826 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.696014 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:13.698986 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:13.698990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:13.699001 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:13.699027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:13.700666 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701191 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:13.701198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:13.701208 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:13.701317 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701641 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:13.701646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:13.701648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:13.702033 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:13.702174 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:13.702771 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 02:11:13.703047 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260806 02:11:13.703907 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.689586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:44907 (local address 127.28.21.1:49959)
0806 02:11:13.689989 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:13.690004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:13.690127 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:13.691701 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:13.691707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:13.692255 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:13.692703 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:13.692718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.694206 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.694209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:13.695244 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:13.695254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:13.695957 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:13.695961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:13.696187 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:13.696891 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:13.696902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:13.698791 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:13.700792 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:13.700795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:13.700797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:13.701074 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:13.701418 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:13.701422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:13.701424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:13.701544 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:13.702155 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:13.702158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:13.702397 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 02:11:13.702649 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 02:11:13.703095 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":396,"spinlock_wait_cycles":17024}
W20260806 02:11:13.705917 13880 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:11:13.706594 13879 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:11:13.706672 13879 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786; no voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c
I20260806 02:11:13.706892 14300 raft_consensus.cc:2726] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:11:13.765026 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.764602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:34137)
0806 02:11:13.764859 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:13.764915 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260806 02:11:13.960626 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:13.960206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:45579)
0806 02:11:13.960474 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:13.960530 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260806 02:11:14.618230 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:14.617628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:39267)
0806 02:11:14.617959 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:14.618055 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260806 02:11:14.766320 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:14.765859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:49785)
0806 02:11:14.766147 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:14.766202 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260806 02:11:14.961823 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:14.961394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:59659)
0806 02:11:14.961661 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:14.961714 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260806 02:11:15.619460 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:15.618990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:58587)
0806 02:11:15.619240 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:15.619306 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260806 02:11:15.619820 14310 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:15.619932 14310 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:15.620373 14310 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
W20260806 02:11:15.621097 13879 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:11:15.621181 13880 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:11:15.621376 13880 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786; no voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c
I20260806 02:11:15.621562 14310 raft_consensus.cc:2726] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:11:15.767578 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:15.767055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:53943)
0806 02:11:15.767417 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:15.767479 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260806 02:11:15.962996 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:15.962548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:60281)
0806 02:11:15.962842 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:15.962898 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260806 02:11:16.620759 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:16.620310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:41797)
0806 02:11:16.620587 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:16.620655 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260806 02:11:16.768716 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:16.768319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:35815)
0806 02:11:16.768566 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:16.768620 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260806 02:11:16.964303 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:16.963881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:36587)
0806 02:11:16.964152 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:16.964209 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260806 02:11:17.622484 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:17.621934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:34749)
0806 02:11:17.622279 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:17.622350 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
I20260806 02:11:17.770171 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:17.769690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:54733)
0806 02:11:17.769987 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:17.770058 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260806 02:11:17.965718 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:17.965235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:58639)
0806 02:11:17.965551 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:17.965619 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
I20260806 02:11:18.452688 14319 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:18.452831 14319 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:18.453341 14319 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
W20260806 02:11:18.454202 13879 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:11:18.454309 13880 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:11:18.454453 13880 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786; no voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c
I20260806 02:11:18.454645 14319 raft_consensus.cc:2726] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:11:18.623764 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:18.623215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:60189)
0806 02:11:18.623604 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:18.623662 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":77,"threads_started":1}
I20260806 02:11:18.771514 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:18.771074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:45449)
0806 02:11:18.771348 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:18.771409 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260806 02:11:18.967100 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:18.966593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:47125)
0806 02:11:18.966872 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:18.966930 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260806 02:11:19.625032 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:19.624591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:56755)
0806 02:11:19.624863 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:19.624918 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260806 02:11:19.772706 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:19.772272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:39529)
0806 02:11:19.772560 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:19.772613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260806 02:11:19.968436 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:19.967937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:42619)
0806 02:11:19.968282 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:19.968339 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":77,"threads_started":1}
I20260806 02:11:20.626184 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:20.625777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:37423)
0806 02:11:20.626026 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:20.626084 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260806 02:11:20.773867 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:20.773464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:60757)
0806 02:11:20.773716 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:20.773772 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260806 02:11:20.969750 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:20.969253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:59981)
0806 02:11:20.969596 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:20.969654 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
I20260806 02:11:21.627429 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:21.626949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:39313)
0806 02:11:21.627258 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:21.627330 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260806 02:11:21.775239 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:21.774716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:40053)
0806 02:11:21.774997 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:21.775122 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260806 02:11:21.971099 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:21.970633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:58575)
0806 02:11:21.970936 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:21.970998 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260806 02:11:22.172796 14335 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:22.172940 14335 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:22.173396 14335 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
W20260806 02:11:22.174309 13880 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:11:22.174504 13879 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:11:22.174582 13879 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786; no voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c
I20260806 02:11:22.174707 14335 raft_consensus.cc:2726] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:11:22.628744 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:22.628366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:38135)
0806 02:11:22.628601 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:22.628651 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260806 02:11:22.776497 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:22.776072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:36841)
0806 02:11:22.776328 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:22.776396 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260806 02:11:22.972482 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:22.971985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:58191)
0806 02:11:22.972277 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:22.972349 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260806 02:11:23.630005 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:23.629586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:40133)
0806 02:11:23.629844 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:23.629902 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260806 02:11:23.777740 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:23.777289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:59235)
0806 02:11:23.777575 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:23.777638 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260806 02:11:23.973670 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:23.973223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:37879)
0806 02:11:23.973515 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:23.973572 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
I20260806 02:11:24.631335 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:24.630870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:42087)
0806 02:11:24.631168 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:24.631228 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
I20260806 02:11:24.779072 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:24.778638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:57699)
0806 02:11:24.778902 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:24.778962 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260806 02:11:24.898546 14344 raft_consensus.cc:482] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:24.898710 14344 raft_consensus.cc:504] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6beb872c50643f4909d395beba78c4c" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 44907 } } peers { permanent_uuid: "5f7eba7b375c463da0b4dbba11e4f786" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 42685 } } peers { permanent_uuid: "a318f3b366674118a30bef8093f3bc90" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36395 } }
I20260806 02:11:24.899149 14344 leader_election.cc:288] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907), a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395)
W20260806 02:11:24.900033 13879 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6beb872c50643f4909d395beba78c4c (127.28.21.3:44907): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
W20260806 02:11:24.900200 13880 leader_election.cc:341] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a318f3b366674118a30bef8093f3bc90 (127.28.21.2:36395): Not found: Tablet not found: 3ab5bb5bdd9e4336a4014b36e8cceeda
I20260806 02:11:24.900279 13880 leader_election.cc:302] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f7eba7b375c463da0b4dbba11e4f786; no voters: a318f3b366674118a30bef8093f3bc90, b6beb872c50643f4909d395beba78c4c
I20260806 02:11:24.900494 14344 raft_consensus.cc:2726] T 3ab5bb5bdd9e4336a4014b36e8cceeda P 5f7eba7b375c463da0b4dbba11e4f786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:11:24.974918 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:24.974485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:56789)
0806 02:11:24.974758 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:24.974818 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260806 02:11:25.632511 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:25.632106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:40321)
0806 02:11:25.632351 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:25.632407 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260806 02:11:25.780234 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:25.779816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:53377)
0806 02:11:25.780072 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:25.780132 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260806 02:11:25.976248 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:25.975713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:53015)
0806 02:11:25.976013 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:25.976143 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260806 02:11:26.633838 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:26.633352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:39165)
0806 02:11:26.633643 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:26.633720 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
I20260806 02:11:26.781692 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:26.781179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:58093)
0806 02:11:26.781526 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:26.781593 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":151,"threads_started":1}
I20260806 02:11:26.977593 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:26.977153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:40979)
0806 02:11:26.977418 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:26.977489 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260806 02:11:27.635478 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:27.634966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.1:56925)
0806 02:11:27.635275 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:27.635379 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260806 02:11:27.783097 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:27.782631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.2:56173)
0806 02:11:27.782917 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:27.782981 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
I20260806 02:11:27.972608 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 13821
I20260806 02:11:27.979091 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:27.978639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:44115 (local address 127.28.21.3:38405)
0806 02:11:27.978876 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:27.978959 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.62:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260806 02:11:27.979377 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14008
I20260806 02:11:27.986152 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14158
2026-08-06T02:11:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59441 ms)
[----------] 30 tests from SecurityITest (244139 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:28 dist-test-slave-gp6n krb5kdc[14357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:28 dist-test-slave-gp6n krb5kdc[14357](info): set up 2 sockets
Aug 06 02:11:28 dist-test-slave-gp6n krb5kdc[14357](info): commencing operation
krb5kdc: starting...
W20260806 02:11:29.062579 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:11:29.194252 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42915
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:29.305562 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:29.305866 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:29.305941 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:29.309892 14371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:29.309974 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:29.310012 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:29.310065 14371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:29.310104 14371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:29.315243 14371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:29.316262 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:29.317276 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:29.325603 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.325603 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.325613 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:29.326025 14371 server_base.cc:1034] running on GCE node
I20260806 02:11:29.326612 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:29.327651 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:29.328812 14371 hybrid_clock.cc:648] HybridClock initialized: now 1785982289328796 us; error 24 us; skew 500 ppm
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:29.331714 14371 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:29.332633 14371 webserver.cc:460] Webserver started at http://127.28.21.62:35505/ using document root <none> and password file <none>
I20260806 02:11:29.333209 14371 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:29.333299 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:29.333544 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:29.335665 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "b814945db3a040318c99897c2d5d90a4"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "8dc8dff703ce2db44ce5c03285fb5053"
server_key_iv: "3838e58c5c2c7d782c5391c9c7f3e162"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.336324 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "b814945db3a040318c99897c2d5d90a4"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "8dc8dff703ce2db44ce5c03285fb5053"
server_key_iv: "3838e58c5c2c7d782c5391c9c7f3e162"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.340183 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:11:29.342654 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.343796 14371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:29.343938 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "b814945db3a040318c99897c2d5d90a4"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "8dc8dff703ce2db44ce5c03285fb5053"
server_key_iv: "3838e58c5c2c7d782c5391c9c7f3e162"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.344040 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:29.354544 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:29.357203 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:29.357396 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:29.364449 14371 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:42915
I20260806 02:11:29.364480 14441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:42915 every 8 connection(s)
I20260806 02:11:29.365408 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:29.368340 14442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:29.370801 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14371
I20260806 02:11:29.370951 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:29.371268 28756 external_mini_cluster.cc:1428] Setting key a7e2f5dd29e4079e66cfea18afd17a79
I20260806 02:11:29.374902 14442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: Bootstrap starting.
I20260806 02:11:29.377568 14442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:29.378365 14442 log.cc:824] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:29.380259 14442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: No bootstrap required, opened a new log
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:29.383524 14442 raft_consensus.cc:348] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } }
I20260806 02:11:29.383807 14442 raft_consensus.cc:374] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:29.383883 14442 raft_consensus.cc:729] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b814945db3a040318c99897c2d5d90a4, State: Initialized, Role: FOLLOWER
I20260806 02:11:29.384348 14442 consensus_queue.cc:260] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } }
I20260806 02:11:29.384502 14442 raft_consensus.cc:388] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:29.384572 14442 raft_consensus.cc:482] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:29.384667 14442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:29.385540 14442 raft_consensus.cc:504] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } }
I20260806 02:11:29.385877 14442 leader_election.cc:302] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b814945db3a040318c99897c2d5d90a4; no voters:
I20260806 02:11:29.386186 14442 leader_election.cc:288] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:29.386569 14449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:29.386798 14449 raft_consensus.cc:686] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [term 1 LEADER]: Becoming Leader. State: Replica: b814945db3a040318c99897c2d5d90a4, State: Running, Role: LEADER
I20260806 02:11:29.387157 14449 consensus_queue.cc:237] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } }
I20260806 02:11:29.387511 14442 sys_catalog.cc:564] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:29.387837 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:29.372853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.28.21.62:42915)
0806 02:11:29.373319 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:29.373338 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:29.373382 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:11:29.374207 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:29.374231 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:29.374672 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:29.375054 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:29.375777 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.377274 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.378203 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.378931 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.379107 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:29.381610 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:29.381615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:29.381628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:29.381653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:29.384411 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.384936 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.384942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.384948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.385079 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.385369 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.385372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.385374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.385832 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:29.386047 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:29.386310 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 02:11:29.386685 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":129,"threads_started":1}
I20260806 02:11:29.388398 14449 sys_catalog.cc:455] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b814945db3a040318c99897c2d5d90a4. Latest consensus state: current_term: 1 leader_uuid: "b814945db3a040318c99897c2d5d90a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } } }
I20260806 02:11:29.388538 14449 sys_catalog.cc:458] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:29.388931 14450 sys_catalog.cc:455] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b814945db3a040318c99897c2d5d90a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b814945db3a040318c99897c2d5d90a4" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42915 } } }
I20260806 02:11:29.390216 14450 sys_catalog.cc:458] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4 [sys.catalog]: This master's current role is: LEADER
W20260806 02:11:29.392237 14464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:29.392328 14464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:29.392403 14458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:29.393066 14458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:29.398319 14458 catalog_manager.cc:1347] Generated new cluster ID: efabf515e8c94219ad17cb2955180943
I20260806 02:11:29.398381 14458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:29.410398 14458 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:29.411640 14458 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:29.420298 14458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: Generated new TSK 0
I20260806 02:11:29.421043 14458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:11:29.449922 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:29.561076 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:29.561353 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:29.561426 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:29.565402 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:29.565490 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:29.565584 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:29.570811 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:29.571894 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:29.572956 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:29.583053 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.583091 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.583040 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:29.583303 14470 server_base.cc:1034] running on GCE node
I20260806 02:11:29.583743 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:29.584403 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:29.585563 14470 hybrid_clock.cc:648] HybridClock initialized: now 1785982289585546 us; error 29 us; skew 500 ppm
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:29.588582 14470 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:11:29.589484 14470 webserver.cc:460] Webserver started at http://127.28.21.1:34115/ using document root <none> and password file <none>
I20260806 02:11:29.590057 14470 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:29.590147 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:29.590386 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:29.592530 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "436d74be04784b79ab9ae17376b55dd9"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "de2e90313fa70d7ddcd61eb99df22282"
server_key_iv: "609af27d450d088c73b0b86647a5f246"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.593101 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "436d74be04784b79ab9ae17376b55dd9"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "de2e90313fa70d7ddcd61eb99df22282"
server_key_iv: "609af27d450d088c73b0b86647a5f246"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.597045 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:11:29.599587 14488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.600917 14470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:29.601050 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "436d74be04784b79ab9ae17376b55dd9"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "de2e90313fa70d7ddcd61eb99df22282"
server_key_iv: "609af27d450d088c73b0b86647a5f246"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.601166 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:29.618693 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:29.621610 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:29.621816 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:29.622454 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:29.623296 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:29.623364 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.623435 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:29.623486 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.632854 14470 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:44855
I20260806 02:11:29.633054 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:44855 every 8 connection(s)
I20260806 02:11:29.633816 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:29.636994 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14470
I20260806 02:11:29.637144 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:29.637531 28756 external_mini_cluster.cc:1428] Setting key f404ba1b158d2757f6fc3493b7d808a8
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:11:29.650326 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:29.635763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:43159 (local address 127.28.21.62:42915)
0806 02:11:29.635912 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:29.635915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:29.637260 (+ 1345us) server_negotiation.cc:407] Connection header received
0806 02:11:29.638559 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:29.638563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:29.638623 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:29.638734 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:29.640486 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.641025 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.642355 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.642656 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.642745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:29.645986 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:29.645993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:29.645999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:29.646040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:29.647956 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.648591 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.648596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.648598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.648707 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.649072 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.649077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.649080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.649344 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:29.649451 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:29.650078 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 02:11:29.650197 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:11:29.651307 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:29.636096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42915 (local address 127.28.21.1:43159)
0806 02:11:29.637053 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:29.637134 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:29.638282 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:29.638920 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:29.638927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:29.639495 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:29.640274 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:29.640299 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.641164 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.641170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:29.642229 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:29.642239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.642834 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.642842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:29.643099 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:29.643791 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:29.643831 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:29.645763 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:29.648110 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:29.648121 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:29.648139 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:29.648478 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:29.648814 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:29.648816 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:29.648817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:29.648971 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:29.649472 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:29.649478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:29.649800 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 02:11:29.649998 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 02:11:29.650292 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:11:29.652781 14608 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42915
I20260806 02:11:29.653047 14608 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:29.653579 14608 heartbeater.cc:511] Master 127.28.21.62:42915 requested a full tablet report, sending...
I20260806 02:11:29.655269 14403 ts_manager.cc:194] Registered new tserver with Master: 436d74be04784b79ab9ae17376b55dd9 (127.28.21.1:44855)
I20260806 02:11:29.656421 14403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:43159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:11:29.670207 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:29.783500 14613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:29.783807 14613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:29.783878 14613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:29.788066 14613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:29.788169 14613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:29.788255 14613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:29.793733 14613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:29.794881 14613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:29.795917 14613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:29.805737 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.805756 14618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:29.805791 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:29.805996 14613 server_base.cc:1034] running on GCE node
I20260806 02:11:29.806667 14613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:29.807288 14613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:29.808499 14613 hybrid_clock.cc:648] HybridClock initialized: now 1785982289808479 us; error 30 us; skew 500 ppm
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:29.811597 14613 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:11:29.812534 14613 webserver.cc:460] Webserver started at http://127.28.21.2:35195/ using document root <none> and password file <none>
I20260806 02:11:29.813139 14613 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:29.813243 14613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:29.813489 14613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:29.815761 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "c91bf38c22874c24b8ba9201e989734f"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "0ad541ba8e7fb9343a1064c0e25366b9"
server_key_iv: "0194240f405690e3668b4cf6be67708d"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.816404 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "c91bf38c22874c24b8ba9201e989734f"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "0ad541ba8e7fb9343a1064c0e25366b9"
server_key_iv: "0194240f405690e3668b4cf6be67708d"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.820524 14613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:11:29.823189 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.824476 14613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:29.824621 14613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "c91bf38c22874c24b8ba9201e989734f"
format_stamp: "Formatted at 2026-08-06 02:11:29 on dist-test-slave-gp6n"
server_key: "0ad541ba8e7fb9343a1064c0e25366b9"
server_key_iv: "0194240f405690e3668b4cf6be67708d"
server_key_version: "encryptionkey@0"
I20260806 02:11:29.824734 14613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:29.839597 14613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:29.842525 14613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:29.842715 14613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:29.843290 14613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:29.844126 14613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:29.844202 14613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.844291 14613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:29.844337 14613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:29.853648 14613 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:33445
I20260806 02:11:29.853672 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:33445 every 8 connection(s)
I20260806 02:11:29.854697 14613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:29.857522 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14613
I20260806 02:11:29.857686 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:29.857992 28756 external_mini_cluster.cc:1428] Setting key 20ff6b90a455931e103a4eeac8794c93
Aug 06 02:11:29 dist-test-slave-gp6n krb5kdc[14357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982289, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:11:29.870923 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:29.856779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:40209 (local address 127.28.21.62:42915)
0806 02:11:29.856948 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:29.856952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:29.858066 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 02:11:29.859197 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:29.859199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:29.859242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:29.859372 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:29.860690 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.861549 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.863374 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.863593 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.863676 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:29.866890 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:29.866893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:29.866895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:29.866919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:29.868588 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.869233 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.869237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.869238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.869369 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:29.869717 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:29.869719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:29.869720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:29.869913 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:29.870024 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:29.870613 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 02:11:29.870747 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:11:29.871858 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:29.857157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42915 (local address 127.28.21.2:40209)
0806 02:11:29.857869 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:29.857923 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:29.858978 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:29.859553 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:29.859564 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:29.860056 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:29.860543 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:29.860558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.861697 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.861702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:29.863207 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:29.863222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:29.863752 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:29.863758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:29.863995 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:29.864717 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:29.864745 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:29.866695 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:29.868716 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:29.868722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:29.868738 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:29.869115 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:29.869461 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:29.869464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:29.869465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:29.869612 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:29.870025 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:29.870033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:29.870292 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:11:29.870548 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 02:11:29.870899 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":130,"threads_started":1}
I20260806 02:11:29.873062 14749 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42915
I20260806 02:11:29.873325 14749 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:29.873840 14749 heartbeater.cc:511] Master 127.28.21.62:42915 requested a full tablet report, sending...
I20260806 02:11:29.874979 14403 ts_manager.cc:194] Registered new tserver with Master: c91bf38c22874c24b8ba9201e989734f (127.28.21.2:33445)
I20260806 02:11:29.875685 14403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:40209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:11:29.889302 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:30.000720 14755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:30.000969 14755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:30.001019 14755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:30.004969 14755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:30.005028 14755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:30.005108 14755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:30.010380 14755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:30.011384 14755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:30.012413 14755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:30.022298 14763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.022318 14762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.022318 14765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.022372 14755 server_base.cc:1034] running on GCE node
I20260806 02:11:30.022863 14755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:30.023414 14755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:30.024567 14755 hybrid_clock.cc:648] HybridClock initialized: now 1785982290024552 us; error 25 us; skew 500 ppm
Aug 06 02:11:30 dist-test-slave-gp6n krb5kdc[14357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982290, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:30.027314 14755 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:11:30.028236 14755 webserver.cc:460] Webserver started at http://127.28.21.3:35997/ using document root <none> and password file <none>
I20260806 02:11:30.028841 14755 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:30.028927 14755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:30.029142 14755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:30.031177 14755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "53d76715bd4c48acb1b7a6202e635d04"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "213a5c70761b22d3159d3df761c7e5f1"
server_key_iv: "87c933f5dc82d3667f429b08438653a8"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.031735 14755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "53d76715bd4c48acb1b7a6202e635d04"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "213a5c70761b22d3159d3df761c7e5f1"
server_key_iv: "87c933f5dc82d3667f429b08438653a8"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.035362 14755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:30.037693 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.038920 14755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:30.039049 14755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "53d76715bd4c48acb1b7a6202e635d04"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "213a5c70761b22d3159d3df761c7e5f1"
server_key_iv: "87c933f5dc82d3667f429b08438653a8"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.039167 14755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:30.050843 14755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:30.053998 14755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:30.054194 14755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:30.054822 14755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:30.055629 14755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:30.055694 14755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.055753 14755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:30.055800 14755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.064711 14755 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:46503
I20260806 02:11:30.064738 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:46503 every 8 connection(s)
I20260806 02:11:30.065601 14755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:30.066241 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14755
I20260806 02:11:30.066391 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:30.066762 28756 external_mini_cluster.cc:1428] Setting key 0b10765a5c3108f93fb717dd4bedcfdb
Aug 06 02:11:30 dist-test-slave-gp6n krb5kdc[14357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982290, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:30.079237 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.067654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:39697 (local address 127.28.21.62:42915)
0806 02:11:30.067800 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.067804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.069494 (+ 1690us) server_negotiation.cc:407] Connection header received
0806 02:11:30.070329 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.070332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.070442 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.070523 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:30.071624 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.072154 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.073271 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.073529 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.073624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.076049 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:30.076053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:30.076055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:30.076079 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:30.077382 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:30.077831 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:30.077833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:30.077835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:30.077912 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:30.078216 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:30.078219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:30.078220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:30.078474 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:30.078567 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.078999 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.079102 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:11:30.079993 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.068214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42915 (local address 127.28.21.3:39697)
0806 02:11:30.069350 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.069392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.070141 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.070671 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.070677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.071022 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:30.071480 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.071492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.072265 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.072267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.073137 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.073145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.073665 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.073667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.073812 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.074443 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:30.074468 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:30.075877 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:30.077482 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:30.077486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:30.077497 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:30.077748 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:30.077985 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:30.077987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:30.077988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:30.078114 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:30.078573 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:30.078578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:30.078796 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.078949 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.079188 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":709,"threads_started":1}
I20260806 02:11:30.081270 14892 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42915
I20260806 02:11:30.081512 14892 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:30.081979 14892 heartbeater.cc:511] Master 127.28.21.62:42915 requested a full tablet report, sending...
I20260806 02:11:30.083016 14402 ts_manager.cc:194] Registered new tserver with Master: 53d76715bd4c48acb1b7a6202e635d04 (127.28.21.3:46503)
I20260806 02:11:30.083616 14402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:39697
I20260806 02:11:30.093006 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:30.103152 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.095097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.28.21.62:42915)
0806 02:11:30.095261 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.095265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.095402 (+ 137us) server_negotiation.cc:407] Connection header received
0806 02:11:30.095496 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.095498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.095541 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.095605 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:30.096300 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.096828 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.097716 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.098040 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.098163 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.099868 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:30.099875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:30.099878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:30.099906 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:30.101458 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:30.101882 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:30.101885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:30.101887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:30.101975 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:30.102232 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:30.102235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:30.102236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:30.102554 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:30.102665 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.102921 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.103025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:11:30.106139 14402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:30.109127 14402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:30.124087 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.117308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:33445 (local address 127.0.0.1:46080)
0806 02:11:30.119111 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.119128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.119226 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.119791 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.119794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.119812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:30.119909 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.119916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.121004 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.121008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.122433 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.122448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123472 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.123476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.123592 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.123633 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.123782 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.123919 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"mutex_wait_us":1755,"thread_start_us":114,"threads_started":1}
I20260806 02:11:30.124349 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.117308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44855 (local address 127.0.0.1:38978)
0806 02:11:30.117718 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.117780 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.117911 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.119313 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.119320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.119357 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:30.119574 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.119582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.121096 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.121102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.122803 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.122814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123946 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.123950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.124052 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.124075 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.124127 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.124198 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":363,"spinlock_wait_cycles":6272,"thread_start_us":97,"threads_started":1}
I20260806 02:11:30.124715 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.117605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.28.21.2:33445)
0806 02:11:30.119274 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.119289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.119313 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:11:30.119414 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.119420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.119644 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.119821 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:30.120002 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.120829 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.122573 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123338 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.123472 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.123642 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.123816 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.123929 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":118,"threads_started":1}
I20260806 02:11:30.124922 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.118105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:46503 (local address 127.0.0.1:55566)
0806 02:11:30.118374 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.118387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.118511 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.119313 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.119320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.119356 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:30.119573 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.119584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.121129 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.121132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.122803 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.122814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123946 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.123950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.124567 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.124594 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.124703 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.124773 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":755,"spinlock_wait_cycles":4736,"thread_start_us":124,"threads_started":1}
I20260806 02:11:30.125162 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.118308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.28.21.3:46503)
0806 02:11:30.118770 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.118784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.118807 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:11:30.118891 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.118897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.119122 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.119757 (+ 635us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:30.119916 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.120984 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.122951 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123797 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.123998 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.124556 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.124738 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.125014 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260806 02:11:30.125562 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.118560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.28.21.1:44855)
0806 02:11:30.118901 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.118915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.118934 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:11:30.118997 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.119000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.119143 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.119436 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:30.119698 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.120927 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.122951 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.123797 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.125105 (+ 1308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.125261 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.125314 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.125416 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":144,"threads_started":1}
I20260806 02:11:30.127991 14538 tablet_service.cc:1467] Processing CreateTablet for tablet a62df3deeab04d57b246b6fefecd2c21 (DEFAULT_TABLE table=test-table [id=0ff9451a4b844de8b0d2936f20922778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:30.128031 14820 tablet_service.cc:1467] Processing CreateTablet for tablet a62df3deeab04d57b246b6fefecd2c21 (DEFAULT_TABLE table=test-table [id=0ff9451a4b844de8b0d2936f20922778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:30.128047 14680 tablet_service.cc:1467] Processing CreateTablet for tablet a62df3deeab04d57b246b6fefecd2c21 (DEFAULT_TABLE table=test-table [id=0ff9451a4b844de8b0d2936f20922778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:30.129145 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62df3deeab04d57b246b6fefecd2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:30.129168 14820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62df3deeab04d57b246b6fefecd2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:30.129161 14680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62df3deeab04d57b246b6fefecd2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:30.135339 14911 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: Bootstrap starting.
I20260806 02:11:30.136706 14913 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Bootstrap starting.
I20260806 02:11:30.137620 14912 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Bootstrap starting.
I20260806 02:11:30.137830 14911 tablet_bootstrap.cc:654] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:30.139194 14911 log.cc:824] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:30.140228 14912 tablet_bootstrap.cc:654] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:30.140477 14913 tablet_bootstrap.cc:654] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:30.141047 14912 log.cc:824] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:30.141450 14911 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: No bootstrap required, opened a new log
I20260806 02:11:30.141660 14911 ts_tablet_manager.cc:1397] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:30.141662 14913 log.cc:824] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:30.143179 14912 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: No bootstrap required, opened a new log
I20260806 02:11:30.143442 14913 tablet_bootstrap.cc:492] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: No bootstrap required, opened a new log
I20260806 02:11:30.143594 14912 ts_tablet_manager.cc:1397] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:11:30.143640 14913 ts_tablet_manager.cc:1397] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:11:30.145634 14911 raft_consensus.cc:348] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.145897 14911 raft_consensus.cc:374] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:30.145947 14911 raft_consensus.cc:729] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436d74be04784b79ab9ae17376b55dd9, State: Initialized, Role: FOLLOWER
I20260806 02:11:30.146449 14911 consensus_queue.cc:260] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.147176 14608 heartbeater.cc:503] Master 127.28.21.62:42915 was elected leader, sending a full tablet report...
I20260806 02:11:30.147476 14911 ts_tablet_manager.cc:1428] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:11:30.147464 14912 raft_consensus.cc:348] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.147774 14912 raft_consensus.cc:374] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:30.147853 14912 raft_consensus.cc:729] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91bf38c22874c24b8ba9201e989734f, State: Initialized, Role: FOLLOWER
I20260806 02:11:30.148120 14913 raft_consensus.cc:348] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.148272 14912 consensus_queue.cc:260] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.148365 14913 raft_consensus.cc:374] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:30.148417 14913 raft_consensus.cc:729] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d76715bd4c48acb1b7a6202e635d04, State: Initialized, Role: FOLLOWER
I20260806 02:11:30.148836 14913 consensus_queue.cc:260] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.148999 14749 heartbeater.cc:503] Master 127.28.21.62:42915 was elected leader, sending a full tablet report...
I20260806 02:11:30.149256 14912 ts_tablet_manager.cc:1428] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:11:30.149678 14913 ts_tablet_manager.cc:1428] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:11:30.149960 14892 heartbeater.cc:503] Master 127.28.21.62:42915 was elected leader, sending a full tablet report...
I20260806 02:11:30.213052 14920 raft_consensus.cc:482] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:30.213246 14920 raft_consensus.cc:504] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.214318 14920 leader_election.cc:288] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 436d74be04784b79ab9ae17376b55dd9 (127.28.21.1:44855), c91bf38c22874c24b8ba9201e989734f (127.28.21.2:33445)
I20260806 02:11:30.219265 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.214555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:33445 (local address 127.28.21.3:58259)
0806 02:11:30.214688 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.214706 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.214807 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.215068 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.215071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.215089 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:30.215220 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.215226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.215964 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.215967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.217321 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.217347 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.218734 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.218739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.218881 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.218915 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.219000 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.219095 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:11:30.219494 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.214555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44855 (local address 127.28.21.3:36205)
0806 02:11:30.215120 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.215131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.215249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.215514 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.215517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.215533 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:30.215643 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.215648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.216604 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.216607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.217905 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.217916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.219039 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.219043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.219167 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.219194 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.219271 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.219337 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":207,"thread_start_us":152,"threads_started":1}
I20260806 02:11:30.219578 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.214660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:36205 (local address 127.28.21.1:44855)
0806 02:11:30.214813 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.214818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.215211 (+ 393us) server_negotiation.cc:407] Connection header received
0806 02:11:30.215357 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.215360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.215406 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.215491 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:30.215748 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.216474 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.218152 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.218791 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.218915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.218977 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.219306 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.219409 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:11:30.219812 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.214702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:58259 (local address 127.28.21.2:33445)
0806 02:11:30.214849 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.214854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.214870 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:11:30.214929 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.214932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.214971 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.215057 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:30.215325 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.215845 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.217482 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.218541 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.218720 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.218842 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.219567 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.219662 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:30.220469 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62df3deeab04d57b246b6fefecd2c21" candidate_uuid: "53d76715bd4c48acb1b7a6202e635d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436d74be04784b79ab9ae17376b55dd9" is_pre_election: true
I20260806 02:11:30.220489 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62df3deeab04d57b246b6fefecd2c21" candidate_uuid: "53d76715bd4c48acb1b7a6202e635d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91bf38c22874c24b8ba9201e989734f" is_pre_election: true
I20260806 02:11:30.220816 14559 raft_consensus.cc:2454] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d76715bd4c48acb1b7a6202e635d04 in term 0.
I20260806 02:11:30.220835 14701 raft_consensus.cc:2454] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d76715bd4c48acb1b7a6202e635d04 in term 0.
I20260806 02:11:30.221366 14773 leader_election.cc:302] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 436d74be04784b79ab9ae17376b55dd9, 53d76715bd4c48acb1b7a6202e635d04, c91bf38c22874c24b8ba9201e989734f; no voters:
I20260806 02:11:30.221876 14920 raft_consensus.cc:2781] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:30.221977 14920 raft_consensus.cc:482] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:30.222041 14920 raft_consensus.cc:3037] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:30.223511 14920 raft_consensus.cc:504] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.223905 14920 leader_election.cc:288] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [CANDIDATE]: Term 1 election: Requested vote from peers 436d74be04784b79ab9ae17376b55dd9 (127.28.21.1:44855), c91bf38c22874c24b8ba9201e989734f (127.28.21.2:33445)
I20260806 02:11:30.224269 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62df3deeab04d57b246b6fefecd2c21" candidate_uuid: "53d76715bd4c48acb1b7a6202e635d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91bf38c22874c24b8ba9201e989734f"
I20260806 02:11:30.224265 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62df3deeab04d57b246b6fefecd2c21" candidate_uuid: "53d76715bd4c48acb1b7a6202e635d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436d74be04784b79ab9ae17376b55dd9"
I20260806 02:11:30.224407 14559 raft_consensus.cc:3037] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:30.224437 14701 raft_consensus.cc:3037] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:30.225725 14559 raft_consensus.cc:2454] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d76715bd4c48acb1b7a6202e635d04 in term 1.
I20260806 02:11:30.225871 14701 raft_consensus.cc:2454] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d76715bd4c48acb1b7a6202e635d04 in term 1.
I20260806 02:11:30.226154 14775 leader_election.cc:302] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436d74be04784b79ab9ae17376b55dd9, 53d76715bd4c48acb1b7a6202e635d04; no voters:
I20260806 02:11:30.226370 14920 raft_consensus.cc:2781] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:30.226619 14920 raft_consensus.cc:686] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 1 LEADER]: Becoming Leader. State: Replica: 53d76715bd4c48acb1b7a6202e635d04, State: Running, Role: LEADER
I20260806 02:11:30.226938 14920 consensus_queue.cc:237] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } }
I20260806 02:11:30.230042 14403 catalog_manager.cc:5495] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53d76715bd4c48acb1b7a6202e635d04 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "53d76715bd4c48acb1b7a6202e635d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53d76715bd4c48acb1b7a6202e635d04" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 46503 } health_report { overall_health: HEALTHY } } }
I20260806 02:11:30.244628 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.240647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.28.21.3:46503)
0806 02:11:30.240781 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.240787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.240855 (+ 68us) server_negotiation.cc:407] Connection header received
0806 02:11:30.241055 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.241058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.241113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.241228 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:30.241486 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.242037 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.243106 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.243342 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.243478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.243789 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:30.244199 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:30.244335 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.244402 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.244464 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:30.250340 14701 raft_consensus.cc:1261] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 1 FOLLOWER]: Refusing update from remote peer 53d76715bd4c48acb1b7a6202e635d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:30.250347 14559 raft_consensus.cc:1261] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 1 FOLLOWER]: Refusing update from remote peer 53d76715bd4c48acb1b7a6202e635d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:30.251065 14920 consensus_queue.cc:1035] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436d74be04784b79ab9ae17376b55dd9" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:30.251251 14922 consensus_queue.cc:1035] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91bf38c22874c24b8ba9201e989734f" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:30.259474 14928 mvcc.cc:205] Tried to move back new op lower bound from 7315383460858560512 to 7315383460772265984. Current Snapshot: MvccSnapshot[applied={T|T < 7315383460858560512}]
I20260806 02:11:30.259913 14927 mvcc.cc:205] Tried to move back new op lower bound from 7315383460858560512 to 7315383460772265984. Current Snapshot: MvccSnapshot[applied={T|T < 7315383460858560512}]
I20260806 02:11:30.266664 14401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:30.266873 14401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:30.269398 14401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b814945db3a040318c99897c2d5d90a4: Sending DeleteTablet for 3 replicas of tablet a62df3deeab04d57b246b6fefecd2c21
I20260806 02:11:30.270273 14538 tablet_service.cc:1514] Processing DeleteTablet for tablet a62df3deeab04d57b246b6fefecd2c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:38978
I20260806 02:11:30.270277 14680 tablet_service.cc:1514] Processing DeleteTablet for tablet a62df3deeab04d57b246b6fefecd2c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:46080
I20260806 02:11:30.270606 14820 tablet_service.cc:1514] Processing DeleteTablet for tablet a62df3deeab04d57b246b6fefecd2c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:30 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:55566
I20260806 02:11:30.270841 14940 tablet_replica.cc:330] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9: stopping tablet replica
I20260806 02:11:30.271095 14941 tablet_replica.cc:330] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: stopping tablet replica
I20260806 02:11:30.271214 14940 raft_consensus.cc:2229] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:30.271431 14941 raft_consensus.cc:2229] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:11:30.271550 14940 raft_consensus.cc:2258] T a62df3deeab04d57b246b6fefecd2c21 P 436d74be04784b79ab9ae17376b55dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:30.271777 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14470
I20260806 02:11:30.274842 14941 raft_consensus.cc:2258] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:30.277068 14941 ts_tablet_manager.cc:1905] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:30.277755 14939 tablet_replica.cc:330] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: stopping tablet replica
I20260806 02:11:30.278065 14939 raft_consensus.cc:2229] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:30.278360 14939 raft_consensus.cc:2258] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:30.279873 14939 ts_tablet_manager.cc:1905] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:30.281241 14941 ts_tablet_manager.cc:1918] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:30.281420 14941 log.cc:1196] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/a62df3deeab04d57b246b6fefecd2c21
I20260806 02:11:30.281893 14941 ts_tablet_manager.cc:1939] T a62df3deeab04d57b246b6fefecd2c21 P 53d76715bd4c48acb1b7a6202e635d04: Deleting consensus metadata
I20260806 02:11:30.282516 14939 ts_tablet_manager.cc:1918] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:30.282611 14939 log.cc:1196] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/a62df3deeab04d57b246b6fefecd2c21
I20260806 02:11:30.282966 14939 ts_tablet_manager.cc:1939] T a62df3deeab04d57b246b6fefecd2c21 P c91bf38c22874c24b8ba9201e989734f: Deleting consensus metadata
I20260806 02:11:30.283496 14389 catalog_manager.cc:4841] TS 53d76715bd4c48acb1b7a6202e635d04 (127.28.21.3:46503): tablet a62df3deeab04d57b246b6fefecd2c21 (table test-table [id=0ff9451a4b844de8b0d2936f20922778]) successfully deleted
I20260806 02:11:30.284063 14387 catalog_manager.cc:4841] TS c91bf38c22874c24b8ba9201e989734f (127.28.21.2:33445): tablet a62df3deeab04d57b246b6fefecd2c21 (table test-table [id=0ff9451a4b844de8b0d2936f20922778]) successfully deleted
W20260806 02:11:30.284106 14389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:44855 (error 108)
I20260806 02:11:30.284349 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14613
I20260806 02:11:30.286616 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.286276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:44855 (local address 127.0.0.1:38994)
0806 02:11:30.286448 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.286524 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 02:11:30.286787 14389 catalog_manager.cc:4584] TS 436d74be04784b79ab9ae17376b55dd9 (127.28.21.1:44855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a62df3deeab04d57b246b6fefecd2c21: Network error: Client connection negotiation failed: client connection to 127.28.21.1:44855: connect: Connection refused (error 111)
I20260806 02:11:30.297696 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14755
I20260806 02:11:30.304108 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14371
2026-08-06T02:11:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:30Z Disabled control of system clock
I20260806 02:11:30.336897 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36073
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:36073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:30.448182 14946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:30.448437 14946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:30.448478 14946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:30.452523 14946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:30.452577 14946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:30.452592 14946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:30.452610 14946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:30.452625 14946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:30.457214 14946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:36073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:30.458153 14946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:30.459509 14946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:30.468068 14951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.468235 14946 server_base.cc:1034] running on GCE node
W20260806 02:11:30.468068 14954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.468119 14952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.468858 14946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:30.469995 14946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:30.471179 14946 hybrid_clock.cc:648] HybridClock initialized: now 1785982290471152 us; error 44 us; skew 500 ppm
I20260806 02:11:30.478178 14946 webserver.cc:460] Webserver started at http://127.28.21.62:39451/ using document root <none> and password file <none>
I20260806 02:11:30.478819 14946 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:30.478911 14946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:30.479143 14946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:30.481201 14946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "f2c747f016e54a38b94b944f62c97d0d"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "881b92a9c50ab1107ee0f19fb1557d79"
server_key_iv: "8a11599dbe9e514165d4729f7c4fec9a"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.481755 14946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "f2c747f016e54a38b94b944f62c97d0d"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "881b92a9c50ab1107ee0f19fb1557d79"
server_key_iv: "8a11599dbe9e514165d4729f7c4fec9a"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.485357 14946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:11:30.487833 14960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.489068 14946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:30.489208 14946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "f2c747f016e54a38b94b944f62c97d0d"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "881b92a9c50ab1107ee0f19fb1557d79"
server_key_iv: "8a11599dbe9e514165d4729f7c4fec9a"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.489323 14946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:30.514791 14946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:30.515532 14946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:30.515730 14946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:30.522996 14946 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:36073
I20260806 02:11:30.523007 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:36073 every 8 connection(s)
I20260806 02:11:30.523926 14946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:30.527060 15016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:30.532521 15016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: Bootstrap starting.
I20260806 02:11:30.533237 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 14946
I20260806 02:11:30.533336 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:30.533622 28756 external_mini_cluster.cc:1428] Setting key a231b883ef209b3a54cadbb59b7f5753
I20260806 02:11:30.535243 15016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:30.535962 15016 log.cc:824] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:30.537809 15016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: No bootstrap required, opened a new log
I20260806 02:11:30.541970 15016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } }
I20260806 02:11:30.542208 15016 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:30.542289 15016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c747f016e54a38b94b944f62c97d0d, State: Initialized, Role: FOLLOWER
I20260806 02:11:30.542742 15016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } }
I20260806 02:11:30.542893 15016 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:30.542975 15016 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:30.543026 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.534758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.28.21.62:36073)
0806 02:11:30.535376 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.535399 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.535433 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:11:30.536046 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.536062 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.536557 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.536916 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:30.537331 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.538328 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.539627 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.540108 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.540223 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.541135 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:30.541140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:30.541151 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:30.541520 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:30.541570 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:30.541590 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:30.541665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.541827 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.542113 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260806 02:11:30.543072 15016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:30.544106 15016 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } }
I20260806 02:11:30.544442 15016 leader_election.cc:302] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c747f016e54a38b94b944f62c97d0d; no voters:
I20260806 02:11:30.544766 15016 leader_election.cc:288] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:30.544831 15021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:30.545022 15021 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [term 1 LEADER]: Becoming Leader. State: Replica: f2c747f016e54a38b94b944f62c97d0d, State: Running, Role: LEADER
I20260806 02:11:30.545632 15021 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } }
I20260806 02:11:30.546003 15016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:30.547292 15022 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c747f016e54a38b94b944f62c97d0d. Latest consensus state: current_term: 1 leader_uuid: "f2c747f016e54a38b94b944f62c97d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } } }
I20260806 02:11:30.547430 15022 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:30.547631 15023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c747f016e54a38b94b944f62c97d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c747f016e54a38b94b944f62c97d0d" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36073 } } }
I20260806 02:11:30.547796 15023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d [sys.catalog]: This master's current role is: LEADER
W20260806 02:11:30.550848 15037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:30.550930 15037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:30.551002 15030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:30.551687 15030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:30.556941 15030 catalog_manager.cc:1347] Generated new cluster ID: 17c6a53a73ca4922b3f43ccc687059aa
I20260806 02:11:30.557008 15030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:30.575433 15030 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:30.576627 15030 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:30.586787 15030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: Generated new TSK 0
I20260806 02:11:30.587602 15030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:30.596315 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--builtin_ntp_servers=127.28.21.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:30.708117 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:30.708385 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:30.708457 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:30.712957 15041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:30.713028 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:30.713107 15041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:30.717779 15041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:30.718873 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:30.720201 15041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:30.729732 15046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.729745 15047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.729745 15049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.729797 15041 server_base.cc:1034] running on GCE node
I20260806 02:11:30.730309 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:30.730984 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:30.732146 15041 hybrid_clock.cc:648] HybridClock initialized: now 1785982290732114 us; error 43 us; skew 500 ppm
I20260806 02:11:30.734021 15041 webserver.cc:460] Webserver started at http://127.28.21.1:44503/ using document root <none> and password file <none>
I20260806 02:11:30.734651 15041 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:30.734715 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:30.734939 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:30.736918 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "82b045e13e314b7b9b1f98472407074b"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "73a1ca8d4c0425cc67492dfc1d833b14"
server_key_iv: "8021988fd697d5c5c3141a42c2e28202"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.737464 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "82b045e13e314b7b9b1f98472407074b"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "73a1ca8d4c0425cc67492dfc1d833b14"
server_key_iv: "8021988fd697d5c5c3141a42c2e28202"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.741004 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:11:30.743515 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.744661 15041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:30.744788 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "82b045e13e314b7b9b1f98472407074b"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "73a1ca8d4c0425cc67492dfc1d833b14"
server_key_iv: "8021988fd697d5c5c3141a42c2e28202"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.744894 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:30.756347 15041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:30.756958 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:30.757144 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:30.757687 15041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:30.758638 15041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:30.758702 15041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.758765 15041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:30.758809 15041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.767686 15041 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:35991
I20260806 02:11:30.767719 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:35991 every 8 connection(s)
I20260806 02:11:30.768574 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:30.773545 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15041
I20260806 02:11:30.773649 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:30.773962 28756 external_mini_cluster.cc:1428] Setting key 598be0a7662e0fe64d6307d637a9113e
I20260806 02:11:30.775889 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--builtin_ntp_servers=127.28.21.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:30.778199 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.770459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:40447 (local address 127.28.21.62:36073)
0806 02:11:30.770600 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.770604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.771608 (+ 1004us) server_negotiation.cc:407] Connection header received
0806 02:11:30.773009 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.773013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.773071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.773173 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:30.774166 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.774852 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.776035 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.776458 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.776589 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.777709 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:30.777714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:30.777716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:30.777726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:30.777746 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:30.777755 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:30.777828 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.777999 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.778060 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:30.779220 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.770828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36073 (local address 127.28.21.1:40447)
0806 02:11:30.771455 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.771496 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.772782 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.773344 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.773351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.773435 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:30.774012 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.774035 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.774981 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.774985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.775880 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.775889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.776618 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.776621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.776780 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.777437 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:30.777439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:30.777490 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:30.777552 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:30.777842 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:30.777854 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:30.777864 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.777997 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.778281 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260806 02:11:30.780553 15177 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36073
I20260806 02:11:30.780849 15177 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:30.781522 15177 heartbeater.cc:511] Master 127.28.21.62:36073 requested a full tablet report, sending...
I20260806 02:11:30.783416 14977 ts_manager.cc:194] Registered new tserver with Master: 82b045e13e314b7b9b1f98472407074b (127.28.21.1:35991)
I20260806 02:11:30.785235 14977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.1:40447
W20260806 02:11:30.889169 15180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:30.889449 15180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:30.889488 15180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:30.893580 15180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:30.893662 15180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:30.893721 15180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:30.898471 15180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:30.899549 15180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:30.900911 15180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:30.910311 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.910508 15180 server_base.cc:1034] running on GCE node
W20260806 02:11:30.910298 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:30.910658 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:30.911028 15180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:30.911616 15180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:30.912775 15180 hybrid_clock.cc:648] HybridClock initialized: now 1785982290912754 us; error 32 us; skew 500 ppm
I20260806 02:11:30.914703 15180 webserver.cc:460] Webserver started at http://127.28.21.2:42765/ using document root <none> and password file <none>
I20260806 02:11:30.915271 15180 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:30.915333 15180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:30.915580 15180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:30.917575 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "a2768506fa5044e6aa17fc10c10d760e"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "30fd98af7441c1dc9f0b118308c87148"
server_key_iv: "64149c696677e14776dbe3625f1a369c"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.918133 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "a2768506fa5044e6aa17fc10c10d760e"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "30fd98af7441c1dc9f0b118308c87148"
server_key_iv: "64149c696677e14776dbe3625f1a369c"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.921676 15180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:11:30.924134 15195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.925414 15180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:30.925547 15180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "a2768506fa5044e6aa17fc10c10d760e"
format_stamp: "Formatted at 2026-08-06 02:11:30 on dist-test-slave-gp6n"
server_key: "30fd98af7441c1dc9f0b118308c87148"
server_key_iv: "64149c696677e14776dbe3625f1a369c"
server_key_version: "encryptionkey@0"
I20260806 02:11:30.925660 15180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:30.945149 15180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:30.945798 15180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:30.946025 15180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:30.946682 15180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:30.947742 15180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:30.947804 15180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.947870 15180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:30.947901 15180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:30.957015 15180 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:35961
I20260806 02:11:30.957047 15316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:35961 every 8 connection(s)
I20260806 02:11:30.957906 15180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:30.963586 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15180
I20260806 02:11:30.963706 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:30.963981 28756 external_mini_cluster.cc:1428] Setting key 1ad7b2855e6bebf6b5213ba922e25b62
I20260806 02:11:30.966038 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--builtin_ntp_servers=127.28.21.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:30.967640 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.959783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:52687 (local address 127.28.21.62:36073)
0806 02:11:30.959927 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:30.959940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:30.961086 (+ 1146us) server_negotiation.cc:407] Connection header received
0806 02:11:30.962476 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:30.962478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:30.962524 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:30.962590 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:30.963494 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.964186 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.965567 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.965770 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.965848 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.967157 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:30.967160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:30.967162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:30.967174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:30.967193 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:30.967201 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:30.967268 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:30.967470 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 02:11:30.967529 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:11:30.968578 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:30.960220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36073 (local address 127.28.21.2:52687)
0806 02:11:30.960947 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:30.960987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:30.962236 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:30.962748 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:30.962754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:30.962837 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:30.963352 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.963372 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.964314 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.964318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.965442 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:30.965453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:30.965928 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:30.965935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:30.966154 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:30.966828 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:30.966833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:30.966889 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:30.966974 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:30.967289 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:30.967300 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:30.967309 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:11:30.967438 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:11:30.967751 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260806 02:11:30.969707 15317 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36073
I20260806 02:11:30.969942 15317 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:30.970495 15317 heartbeater.cc:511] Master 127.28.21.62:36073 requested a full tablet report, sending...
I20260806 02:11:30.971558 14977 ts_manager.cc:194] Registered new tserver with Master: a2768506fa5044e6aa17fc10c10d760e (127.28.21.2:35961)
I20260806 02:11:30.972128 14977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.2:52687
W20260806 02:11:31.077369 15320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:31.077636 15320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:31.077706 15320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:31.081648 15320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:31.081722 15320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:31.081807 15320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:31.086444 15320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:31.087472 15320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:31.088840 15320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:31.098326 15328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:31.098309 15326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:31.098452 15320 server_base.cc:1034] running on GCE node
W20260806 02:11:31.098309 15325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:31.099037 15320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:31.099674 15320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:31.100834 15320 hybrid_clock.cc:648] HybridClock initialized: now 1785982291100814 us; error 34 us; skew 500 ppm
I20260806 02:11:31.103365 15320 webserver.cc:460] Webserver started at http://127.28.21.3:39669/ using document root <none> and password file <none>
I20260806 02:11:31.104211 15320 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:31.104326 15320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:31.104576 15320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:31.106853 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "4cf0f2d5dc364d03a62eff82b9abea51"
format_stamp: "Formatted at 2026-08-06 02:11:31 on dist-test-slave-gp6n"
server_key: "84e6f0cdc3a7d6188956f50173a801f1"
server_key_iv: "173416c1017edfedb86002e247a5f683"
server_key_version: "encryptionkey@0"
I20260806 02:11:31.107486 15320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "4cf0f2d5dc364d03a62eff82b9abea51"
format_stamp: "Formatted at 2026-08-06 02:11:31 on dist-test-slave-gp6n"
server_key: "84e6f0cdc3a7d6188956f50173a801f1"
server_key_iv: "173416c1017edfedb86002e247a5f683"
server_key_version: "encryptionkey@0"
I20260806 02:11:31.111395 15320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:11:31.113895 15334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:31.115161 15320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:31.115286 15320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "4cf0f2d5dc364d03a62eff82b9abea51"
format_stamp: "Formatted at 2026-08-06 02:11:31 on dist-test-slave-gp6n"
server_key: "84e6f0cdc3a7d6188956f50173a801f1"
server_key_iv: "173416c1017edfedb86002e247a5f683"
server_key_version: "encryptionkey@0"
I20260806 02:11:31.115422 15320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:31.144402 15320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:31.145135 15320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:31.145337 15320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:31.145961 15320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:31.147047 15320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:31.147132 15320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:31.147217 15320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:31.147253 15320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:31.156011 15320 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:38115
I20260806 02:11:31.156041 15456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:38115 every 8 connection(s)
I20260806 02:11:31.156934 15320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:31.164216 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15320
I20260806 02:11:31.164346 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:31.164677 28756 external_mini_cluster.cc:1428] Setting key aeccdae7e98dfc32a37cdf2b59822bdb
I20260806 02:11:31.166932 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.158789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:51083 (local address 127.28.21.62:36073)
0806 02:11:31.158917 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.158921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.160031 (+ 1110us) server_negotiation.cc:407] Connection header received
0806 02:11:31.161618 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.161621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.161678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.161785 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:31.162754 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.163279 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.164494 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.164850 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.164972 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.166439 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:31.166446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:31.166452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:31.166465 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:31.166491 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:31.166500 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:31.166574 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.166731 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.166794 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:11:31.167778 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.159203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36073 (local address 127.28.21.3:51083)
0806 02:11:31.159879 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.159922 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.161377 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.161975 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.161983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.162066 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:31.162606 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.162621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.163407 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.163410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.164292 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.164300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.165090 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.165096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.165342 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.166063 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:31.166065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:31.166119 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:31.166205 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:31.166590 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:31.166599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:31.166605 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.166731 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.167034 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260806 02:11:31.168830 15457 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36073
I20260806 02:11:31.169064 15457 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:31.169582 15457 heartbeater.cc:511] Master 127.28.21.62:36073 requested a full tablet report, sending...
I20260806 02:11:31.170588 14977 ts_manager.cc:194] Registered new tserver with Master: 4cf0f2d5dc364d03a62eff82b9abea51 (127.28.21.3:38115)
I20260806 02:11:31.171151 14977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.3:51083
I20260806 02:11:31.178805 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:31.184469 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.180792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.28.21.62:36073)
0806 02:11:31.180947 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.180951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.181156 (+ 205us) server_negotiation.cc:407] Connection header received
0806 02:11:31.181253 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.181255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.181297 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.181372 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:31.181607 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.182120 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.182980 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.183195 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.183290 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.184033 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:31.184036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:31.184044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:31.184055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:31.184071 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:31.184079 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:31.184136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.184227 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.184274 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:11:31.187354 14977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:31.190608 14977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:31.205243 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.200123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.28.21.1:35991)
0806 02:11:31.200583 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.200597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.200625 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:11:31.200777 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.200782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.200972 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.201117 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:31.201353 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.202364 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.203733 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.204473 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.204604 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.204810 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.204964 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.205072 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":128,"threads_started":1}
I20260806 02:11:31.205262 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.198921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:35961 (local address 127.0.0.1:38342)
0806 02:11:31.199299 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.199353 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.199494 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.200339 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.200344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.200369 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:31.200512 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.200519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.202164 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.202167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.203824 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.203836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.204884 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.204891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.204989 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.205011 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.205096 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.205157 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":187,"thread_start_us":85,"threads_started":1}
I20260806 02:11:31.205257 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.199695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:35991 (local address 127.0.0.1:58494)
0806 02:11:31.200555 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.200569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.200665 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.201139 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.201143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.201161 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:31.201239 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.201245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.202503 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.202507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.203612 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.203622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.204636 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.204639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.204729 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.204761 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.204938 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.205088 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":68,"thread_start_us":94,"threads_started":1}
I20260806 02:11:31.205648 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.199032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.28.21.2:35961)
0806 02:11:31.199823 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.199837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.199864 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:11:31.199936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.199939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.200153 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.200678 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:31.200818 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.202027 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.203963 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.204725 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.204840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.205032 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.205341 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.205458 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":130,"threads_started":1}
I20260806 02:11:31.208339 15247 tablet_service.cc:1467] Processing CreateTablet for tablet a42da2fa79ba4768854875ed681cf181 (DEFAULT_TABLE table=test-table [id=88671160c0bb4c558d9c2bd8115cea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:31.208339 15109 tablet_service.cc:1467] Processing CreateTablet for tablet a42da2fa79ba4768854875ed681cf181 (DEFAULT_TABLE table=test-table [id=88671160c0bb4c558d9c2bd8115cea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:31.209308 15247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42da2fa79ba4768854875ed681cf181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:31.209308 15109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42da2fa79ba4768854875ed681cf181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:31.211457 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.199324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:38115 (local address 127.0.0.1:59940)
0806 02:11:31.200114 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.200127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.200211 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.200945 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.200948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.200964 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:31.201066 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.201074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.202099 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.202104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.203831 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.203837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.210982 (+ 7145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.210986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.211090 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.211115 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.211210 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.211291 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":216,"thread_start_us":96,"threads_started":1}
I20260806 02:11:31.211663 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.199666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.28.21.3:38115)
0806 02:11:31.200515 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.200527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.200549 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:11:31.200629 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.200634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.200821 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.200970 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:31.201196 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.201977 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.205098 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.206283 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.210713 (+ 4430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.210912 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.211263 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.211374 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":120,"threads_started":1}
I20260806 02:11:31.214639 15386 tablet_service.cc:1467] Processing CreateTablet for tablet a42da2fa79ba4768854875ed681cf181 (DEFAULT_TABLE table=test-table [id=88671160c0bb4c558d9c2bd8115cea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:31.215763 15386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42da2fa79ba4768854875ed681cf181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:31.216127 15474 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Bootstrap starting.
I20260806 02:11:31.217392 15475 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Bootstrap starting.
I20260806 02:11:31.218536 15474 tablet_bootstrap.cc:654] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:31.219411 15474 log.cc:824] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:31.219589 15475 tablet_bootstrap.cc:654] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:31.220323 15475 log.cc:824] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:31.221815 15474 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: No bootstrap required, opened a new log
I20260806 02:11:31.222000 15474 ts_tablet_manager.cc:1397] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260806 02:11:31.222940 15475 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: No bootstrap required, opened a new log
I20260806 02:11:31.223124 15475 ts_tablet_manager.cc:1397] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:31.223479 15478 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Bootstrap starting.
I20260806 02:11:31.225488 15478 tablet_bootstrap.cc:654] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:31.226128 15478 log.cc:824] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:31.226532 15474 raft_consensus.cc:348] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.226830 15474 raft_consensus.cc:374] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:31.226898 15474 raft_consensus.cc:729] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2768506fa5044e6aa17fc10c10d760e, State: Initialized, Role: FOLLOWER
I20260806 02:11:31.227205 15475 raft_consensus.cc:348] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.227352 15474 consensus_queue.cc:260] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.227603 15475 raft_consensus.cc:374] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:31.227675 15475 raft_consensus.cc:729] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b045e13e314b7b9b1f98472407074b, State: Initialized, Role: FOLLOWER
I20260806 02:11:31.228081 15478 tablet_bootstrap.cc:492] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: No bootstrap required, opened a new log
I20260806 02:11:31.228253 15478 ts_tablet_manager.cc:1397] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:11:31.228394 15474 ts_tablet_manager.cc:1428] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 02:11:31.228526 15475 consensus_queue.cc:260] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.228762 15317 heartbeater.cc:503] Master 127.28.21.62:36073 was elected leader, sending a full tablet report...
I20260806 02:11:31.229184 15177 heartbeater.cc:503] Master 127.28.21.62:36073 was elected leader, sending a full tablet report...
I20260806 02:11:31.229837 15475 ts_tablet_manager.cc:1428] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:11:31.232617 15478 raft_consensus.cc:348] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.232903 15478 raft_consensus.cc:374] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:31.232971 15478 raft_consensus.cc:729] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf0f2d5dc364d03a62eff82b9abea51, State: Initialized, Role: FOLLOWER
I20260806 02:11:31.233417 15478 consensus_queue.cc:260] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.234020 15457 heartbeater.cc:503] Master 127.28.21.62:36073 was elected leader, sending a full tablet report...
I20260806 02:11:31.234244 15478 ts_tablet_manager.cc:1428] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260806 02:11:31.269795 15178 tablet.cc:2369] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:31.335780 15481 raft_consensus.cc:482] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:31.336031 15481 raft_consensus.cc:504] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.337265 15481 leader_election.cc:288] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2768506fa5044e6aa17fc10c10d760e (127.28.21.2:35961), 4cf0f2d5dc364d03a62eff82b9abea51 (127.28.21.3:38115)
I20260806 02:11:31.341483 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.337462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:35961 (local address 127.28.21.1:60553)
0806 02:11:31.337875 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.337890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.338012 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.338294 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.338297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.338314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:31.338490 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.338500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.339386 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.339389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.340270 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.340279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.341031 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.341035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.341134 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.341160 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.341236 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.341304 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":134,"threads_started":1}
I20260806 02:11:31.341506 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.337617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:60553 (local address 127.28.21.2:35961)
0806 02:11:31.337786 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.337791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.337989 (+ 198us) server_negotiation.cc:407] Connection header received
0806 02:11:31.338122 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.338125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.338187 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.338280 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:31.338629 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.339254 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.340394 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.340912 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.340999 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.341052 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.341267 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.341364 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:11:31.341612 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.337777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:41549 (local address 127.28.21.3:38115)
0806 02:11:31.337894 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.337900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.338126 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:11:31.338261 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.338263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.338315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.338435 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:31.338744 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.339243 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.340610 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.341032 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.341134 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.341192 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.341409 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.341482 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:11:31.341863 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.337649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:38115 (local address 127.28.21.1:41549)
0806 02:11:31.338009 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.338026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:31.338140 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:31.338456 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:31.338460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:31.338482 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:31.338627 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.338634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.339363 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.339368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.340479 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:31.340490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.341160 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.341163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:31.341272 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.341298 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:11:31.341380 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:11:31.341450 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":25,"thread_start_us":76,"threads_started":1}
I20260806 02:11:31.342263 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42da2fa79ba4768854875ed681cf181" candidate_uuid: "82b045e13e314b7b9b1f98472407074b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2768506fa5044e6aa17fc10c10d760e" is_pre_election: true
I20260806 02:11:31.342535 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42da2fa79ba4768854875ed681cf181" candidate_uuid: "82b045e13e314b7b9b1f98472407074b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" is_pre_election: true
I20260806 02:11:31.342607 15268 raft_consensus.cc:2454] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b045e13e314b7b9b1f98472407074b in term 0.
I20260806 02:11:31.342829 15408 raft_consensus.cc:2454] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b045e13e314b7b9b1f98472407074b in term 0.
I20260806 02:11:31.343135 15060 leader_election.cc:302] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b045e13e314b7b9b1f98472407074b, a2768506fa5044e6aa17fc10c10d760e; no voters:
I20260806 02:11:31.343400 15481 raft_consensus.cc:2781] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:31.343490 15481 raft_consensus.cc:482] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:31.343536 15481 raft_consensus.cc:3037] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:31.344555 15481 raft_consensus.cc:504] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.344902 15481 leader_election.cc:288] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [CANDIDATE]: Term 1 election: Requested vote from peers a2768506fa5044e6aa17fc10c10d760e (127.28.21.2:35961), 4cf0f2d5dc364d03a62eff82b9abea51 (127.28.21.3:38115)
I20260806 02:11:31.345247 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42da2fa79ba4768854875ed681cf181" candidate_uuid: "82b045e13e314b7b9b1f98472407074b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2768506fa5044e6aa17fc10c10d760e"
I20260806 02:11:31.345412 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42da2fa79ba4768854875ed681cf181" candidate_uuid: "82b045e13e314b7b9b1f98472407074b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf0f2d5dc364d03a62eff82b9abea51"
I20260806 02:11:31.345418 15268 raft_consensus.cc:3037] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:31.345542 15408 raft_consensus.cc:3037] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:31.346748 15268 raft_consensus.cc:2454] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b045e13e314b7b9b1f98472407074b in term 1.
I20260806 02:11:31.346735 15408 raft_consensus.cc:2454] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b045e13e314b7b9b1f98472407074b in term 1.
I20260806 02:11:31.347173 15058 leader_election.cc:302] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf0f2d5dc364d03a62eff82b9abea51, 82b045e13e314b7b9b1f98472407074b; no voters:
I20260806 02:11:31.347702 15481 raft_consensus.cc:2781] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:31.347970 15481 raft_consensus.cc:686] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [term 1 LEADER]: Becoming Leader. State: Replica: 82b045e13e314b7b9b1f98472407074b, State: Running, Role: LEADER
I20260806 02:11:31.348402 15481 consensus_queue.cc:237] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } }
I20260806 02:11:31.351814 14976 catalog_manager.cc:5495] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b reported cstate change: term changed from 0 to 1, leader changed from <none> to 82b045e13e314b7b9b1f98472407074b (127.28.21.1). New cstate: current_term: 1 leader_uuid: "82b045e13e314b7b9b1f98472407074b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82b045e13e314b7b9b1f98472407074b" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 } health_report { overall_health: UNKNOWN } } }
I20260806 02:11:31.393270 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.389822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.28.21.1:35991)
0806 02:11:31.389974 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:31.389978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:31.389993 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:11:31.390072 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:31.390074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:31.390140 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:31.390232 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:31.390467 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.390990 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.391813 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:31.392032 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:31.392137 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:31.392384 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:31.392894 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:31.392991 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:31.393043 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:11:31.393091 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:11:31.399293 15268 raft_consensus.cc:1261] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 1 FOLLOWER]: Refusing update from remote peer 82b045e13e314b7b9b1f98472407074b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:31.399293 15408 raft_consensus.cc:1261] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 1 FOLLOWER]: Refusing update from remote peer 82b045e13e314b7b9b1f98472407074b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:31.400030 15481 consensus_queue.cc:1035] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf0f2d5dc364d03a62eff82b9abea51" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:31.400244 15485 consensus_queue.cc:1035] T a42da2fa79ba4768854875ed681cf181 P 82b045e13e314b7b9b1f98472407074b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2768506fa5044e6aa17fc10c10d760e" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:11:31.407886 15458 tablet.cc:2369] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:31.407974 15491 mvcc.cc:205] Tried to move back new op lower bound from 7315383465564008448 to 7315383465366224896. Current Snapshot: MvccSnapshot[applied={T|T < 7315383465564008448}]
I20260806 02:11:31.416481 14976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:31.416680 14976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:31.419390 14976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2c747f016e54a38b94b944f62c97d0d: Sending DeleteTablet for 3 replicas of tablet a42da2fa79ba4768854875ed681cf181
I20260806 02:11:31.420279 15247 tablet_service.cc:1514] Processing DeleteTablet for tablet a42da2fa79ba4768854875ed681cf181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:31 UTC) from {username='slave'} at 127.0.0.1:38342
I20260806 02:11:31.420271 15109 tablet_service.cc:1514] Processing DeleteTablet for tablet a42da2fa79ba4768854875ed681cf181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:31 UTC) from {username='slave'} at 127.0.0.1:58494
I20260806 02:11:31.420812 15499 tablet_replica.cc:330] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: stopping tablet replica
I20260806 02:11:31.420814 15386 tablet_service.cc:1514] Processing DeleteTablet for tablet a42da2fa79ba4768854875ed681cf181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:31 UTC) from {username='slave'} at 127.0.0.1:59940
I20260806 02:11:31.421144 15499 raft_consensus.cc:2229] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:31.421433 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15041
I20260806 02:11:31.421464 15499 raft_consensus.cc:2258] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:31.421559 15500 tablet_replica.cc:330] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: stopping tablet replica
I20260806 02:11:31.422538 15500 raft_consensus.cc:2229] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:31.422811 15500 raft_consensus.cc:2258] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:31.424626 15500 ts_tablet_manager.cc:1905] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:31.425755 15499 ts_tablet_manager.cc:1905] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:31.426981 15500 ts_tablet_manager.cc:1918] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:31.427100 15500 log.cc:1196] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/a42da2fa79ba4768854875ed681cf181
I20260806 02:11:31.427440 15500 ts_tablet_manager.cc:1939] T a42da2fa79ba4768854875ed681cf181 P 4cf0f2d5dc364d03a62eff82b9abea51: Deleting consensus metadata
I20260806 02:11:31.428526 14962 catalog_manager.cc:4841] TS 4cf0f2d5dc364d03a62eff82b9abea51 (127.28.21.3:38115): tablet a42da2fa79ba4768854875ed681cf181 (table test-table [id=88671160c0bb4c558d9c2bd8115cea09]) successfully deleted
I20260806 02:11:31.429173 15499 ts_tablet_manager.cc:1918] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:31.429276 15499 log.cc:1196] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/a42da2fa79ba4768854875ed681cf181
I20260806 02:11:31.429626 15499 ts_tablet_manager.cc:1939] T a42da2fa79ba4768854875ed681cf181 P a2768506fa5044e6aa17fc10c10d760e: Deleting consensus metadata
I20260806 02:11:31.431139 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15180
W20260806 02:11:31.431232 14964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:35991 (error 108)
I20260806 02:11:31.431699 14964 catalog_manager.cc:4841] TS a2768506fa5044e6aa17fc10c10d760e (127.28.21.2:35961): tablet a42da2fa79ba4768854875ed681cf181 (table test-table [id=88671160c0bb4c558d9c2bd8115cea09]) successfully deleted
I20260806 02:11:31.435511 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:31.435110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:35991 (local address 127.0.0.1:34514)
0806 02:11:31.435287 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:31.435394 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:35991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:11:31.436658 14964 catalog_manager.cc:4584] TS 82b045e13e314b7b9b1f98472407074b (127.28.21.1:35991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a42da2fa79ba4768854875ed681cf181: Network error: Client connection negotiation failed: client connection to 127.28.21.1:35991: connect: Connection refused (error 111)
I20260806 02:11:31.441408 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15320
I20260806 02:11:31.448669 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 14946
2026-08-06T02:11:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:31 dist-test-slave-gp6n krb5kdc[15505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:31 dist-test-slave-gp6n krb5kdc[15505](info): set up 2 sockets
Aug 06 02:11:31 dist-test-slave-gp6n krb5kdc[15505](info): commencing operation
krb5kdc: starting...
W20260806 02:11:32.552650 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:32 dist-test-slave-gp6n krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:11:32.681170 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39951
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:32.790633 15518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:32.790889 15518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:32.790930 15518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:32.794974 15518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:32.795033 15518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:32.795053 15518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:32.795068 15518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:32.795099 15518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:32.800374 15518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:32.801432 15518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:32.802510 15518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:32.810964 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:32.810964 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:32.810985 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:32.811483 15518 server_base.cc:1034] running on GCE node
I20260806 02:11:32.812143 15518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:32.813373 15518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:32.814558 15518 hybrid_clock.cc:648] HybridClock initialized: now 1785982292814537 us; error 36 us; skew 500 ppm
Aug 06 02:11:32 dist-test-slave-gp6n krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982292, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:32.817510 15518 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:32.818462 15518 webserver.cc:460] Webserver started at http://127.28.21.62:45299/ using document root <none> and password file <none>
I20260806 02:11:32.819020 15518 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:32.819103 15518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:32.819329 15518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:32.821376 15518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "4727a3b15e0f4440a9c301073ee04883"
format_stamp: "Formatted at 2026-08-06 02:11:32 on dist-test-slave-gp6n"
server_key: "802a1a8c7ac2f14a04af06bb1173f793"
server_key_iv: "5df9428cda90851ee49c618f5c128dd3"
server_key_version: "encryptionkey@0"
I20260806 02:11:32.821959 15518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "4727a3b15e0f4440a9c301073ee04883"
format_stamp: "Formatted at 2026-08-06 02:11:32 on dist-test-slave-gp6n"
server_key: "802a1a8c7ac2f14a04af06bb1173f793"
server_key_iv: "5df9428cda90851ee49c618f5c128dd3"
server_key_version: "encryptionkey@0"
I20260806 02:11:32.825793 15518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:11:32.828416 15534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:32.829720 15518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:32.829861 15518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "4727a3b15e0f4440a9c301073ee04883"
format_stamp: "Formatted at 2026-08-06 02:11:32 on dist-test-slave-gp6n"
server_key: "802a1a8c7ac2f14a04af06bb1173f793"
server_key_iv: "5df9428cda90851ee49c618f5c128dd3"
server_key_version: "encryptionkey@0"
I20260806 02:11:32.829975 15518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:32.845656 15518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:32.849246 15518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:32.849470 15518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:32.856598 15518 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:39951
I20260806 02:11:32.856660 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:39951 every 8 connection(s)
I20260806 02:11:32.857651 15518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:32.860474 15591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:32.866113 15591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: Bootstrap starting.
I20260806 02:11:32.867532 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15518
I20260806 02:11:32.867667 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:32.867951 28756 external_mini_cluster.cc:1428] Setting key aa0030a650e8db602e852c913b59ddb9
I20260806 02:11:32.868681 15591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:32.869571 15591 log.cc:824] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:32.871789 15591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: No bootstrap required, opened a new log
I20260806 02:11:32.876178 15591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } }
I20260806 02:11:32.876474 15591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:32.876562 15591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4727a3b15e0f4440a9c301073ee04883, State: Initialized, Role: FOLLOWER
Aug 06 02:11:32 dist-test-slave-gp6n krb5kdc[15505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:32.877023 15591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } }
I20260806 02:11:32.877203 15591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:32.877280 15591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:32.877378 15591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:32.878264 15591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } }
I20260806 02:11:32.878624 15591 leader_election.cc:302] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4727a3b15e0f4440a9c301073ee04883; no voters:
I20260806 02:11:32.878965 15591 leader_election.cc:288] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:32.879050 15596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:32.879225 15596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [term 1 LEADER]: Becoming Leader. State: Replica: 4727a3b15e0f4440a9c301073ee04883, State: Running, Role: LEADER
I20260806 02:11:32.879526 15596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } }
I20260806 02:11:32.880250 15591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:32.881237 15598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4727a3b15e0f4440a9c301073ee04883. Latest consensus state: current_term: 1 leader_uuid: "4727a3b15e0f4440a9c301073ee04883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } } }
I20260806 02:11:32.881356 15598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:32.881742 15597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4727a3b15e0f4440a9c301073ee04883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4727a3b15e0f4440a9c301073ee04883" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39951 } } }
I20260806 02:11:32.881879 15597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:32.882395 15605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:32.883566 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:32.869086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.28.21.62:39951)
0806 02:11:32.869510 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:32.869532 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:32.869576 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:11:32.870298 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:32.870316 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:32.870704 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:32.870996 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:32.871947 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:32.872925 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:32.874145 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:32.874689 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:32.874810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:32.877682 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:32.877687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:32.877700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:32.877726 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:32.879878 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:32.880338 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:32.880346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:32.880355 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:32.880499 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:32.880848 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:32.880852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:32.880855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:32.881451 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:32.881654 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:32.881991 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:11:32.882317 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":144,"threads_started":1}
I20260806 02:11:32.885578 15605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:32.891360 15605 catalog_manager.cc:1347] Generated new cluster ID: 667e33db868f45bcadad411fc5cc625a
I20260806 02:11:32.891433 15605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:32.920393 15605 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:32.921528 15605 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:32.936578 15605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: Generated new TSK 0
I20260806 02:11:32.937460 15605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:11:32.977744 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:33.106858 15617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:33.107110 15617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:33.107152 15617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:33.112629 15617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:33.112694 15617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:33.112752 15617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:33.118099 15617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:33.119153 15617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:33.120111 15617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:33.129803 15623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:33.129880 15622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:33.129822 15627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:33.130167 15617 server_base.cc:1034] running on GCE node
I20260806 02:11:33.130659 15617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:33.131258 15617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:33.132452 15617 hybrid_clock.cc:648] HybridClock initialized: now 1785982293132432 us; error 31 us; skew 500 ppm
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:33.135468 15617 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:11:33.136348 15617 webserver.cc:460] Webserver started at http://127.28.21.1:40793/ using document root <none> and password file <none>
I20260806 02:11:33.136930 15617 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:33.136994 15617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:33.137171 15617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:33.139407 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "0bdd9555711d49a188706d151da2647e"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "d93a74db5a4869f7ac3c23fe57c6ff51"
server_key_iv: "333f439c67a9df68b76b2bc7e22d8a17"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.139966 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "0bdd9555711d49a188706d151da2647e"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "d93a74db5a4869f7ac3c23fe57c6ff51"
server_key_iv: "333f439c67a9df68b76b2bc7e22d8a17"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.144547 15617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:11:33.147264 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.148599 15617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:33.148865 15617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "0bdd9555711d49a188706d151da2647e"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "d93a74db5a4869f7ac3c23fe57c6ff51"
server_key_iv: "333f439c67a9df68b76b2bc7e22d8a17"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.148995 15617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:33.167384 15617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:33.170919 15617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:33.171169 15617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:33.171813 15617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:33.172703 15617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:33.172777 15617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.172864 15617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:33.172909 15617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.182693 15617 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:41677
I20260806 02:11:33.182731 15751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:41677 every 8 connection(s)
I20260806 02:11:33.183650 15617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:33.186062 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15617
I20260806 02:11:33.186211 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:33.186615 28756 external_mini_cluster.cc:1428] Setting key f3105ef1706243dd861609d47decd57b
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:11:33.199234 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.185630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:54627 (local address 127.28.21.62:39951)
0806 02:11:33.185779 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.185783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.187183 (+ 1400us) server_negotiation.cc:407] Connection header received
0806 02:11:33.188217 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.188221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.188278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.188424 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:33.189660 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.190597 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.191886 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.192341 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.192472 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.195243 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:33.195248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:33.195251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:33.195285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:33.196880 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.197425 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.197431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.197433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.197522 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.197941 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.197947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.197949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.198147 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:33.198263 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.198871 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.199030 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:11:33.200074 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.186083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:39951 (local address 127.28.21.1:54627)
0806 02:11:33.186976 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.187050 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.187972 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.188576 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.188583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.188977 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:33.189496 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.189510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.190734 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.190738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.191719 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.191729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.192518 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.192521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.192680 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.193292 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:33.193315 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:33.195069 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:33.197008 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.197014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.197030 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.197315 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.197627 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.197631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.197633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.197840 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.198260 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:33.198268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:33.198602 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.198808 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.199177 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":216,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:11:33.201572 15752 heartbeater.cc:347] Connected to a master server at 127.28.21.62:39951
I20260806 02:11:33.201865 15752 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:33.202571 15752 heartbeater.cc:511] Master 127.28.21.62:39951 requested a full tablet report, sending...
I20260806 02:11:33.204277 15551 ts_manager.cc:194] Registered new tserver with Master: 0bdd9555711d49a188706d151da2647e (127.28.21.1:41677)
I20260806 02:11:33.205595 15551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:54627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:11:33.217969 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:33.334210 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:33.334538 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:33.334641 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:33.338706 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:33.338758 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:33.338846 15757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:33.344017 15757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:33.345074 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:33.346094 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:33.355934 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:33.356067 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:33.355934 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:33.356249 15757 server_base.cc:1034] running on GCE node
I20260806 02:11:33.357017 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:33.357699 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:33.358865 15757 hybrid_clock.cc:648] HybridClock initialized: now 1785982293358843 us; error 30 us; skew 500 ppm
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:33.361894 15757 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:11:33.362898 15757 webserver.cc:460] Webserver started at http://127.28.21.2:36945/ using document root <none> and password file <none>
I20260806 02:11:33.363503 15757 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:33.363569 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:33.363806 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:33.366034 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "6699f7fc1dce40b8a09a119f01be5b06"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "42237370358dcd9e077068b7c6f3a0c1"
server_key_iv: "655fff92ee55cda5994607f39ed229a3"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.366641 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "6699f7fc1dce40b8a09a119f01be5b06"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "42237370358dcd9e077068b7c6f3a0c1"
server_key_iv: "655fff92ee55cda5994607f39ed229a3"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.370513 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:11:33.373023 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.374616 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:33.374751 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "6699f7fc1dce40b8a09a119f01be5b06"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "42237370358dcd9e077068b7c6f3a0c1"
server_key_iv: "655fff92ee55cda5994607f39ed229a3"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.374868 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:33.387975 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:33.391604 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:33.391824 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:33.392524 15757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:33.393435 15757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:33.393486 15757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.393556 15757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:33.393597 15757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.403590 15757 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:44919
I20260806 02:11:33.403621 15889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:44919 every 8 connection(s)
I20260806 02:11:33.404578 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:33.405241 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15757
I20260806 02:11:33.405443 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:33.405795 28756 external_mini_cluster.cc:1428] Setting key 6809595a1fa7e7b42d5a429decd98aeb
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
I20260806 02:11:33.421649 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.406717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:55843 (local address 127.28.21.62:39951)
0806 02:11:33.406874 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.406880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.408133 (+ 1253us) server_negotiation.cc:407] Connection header received
0806 02:11:33.409155 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.409162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.409233 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.409353 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:33.411214 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.412191 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.413561 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.414054 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.414206 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.417384 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:33.417391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:33.417395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:33.417435 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:33.419168 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.419728 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.419732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.419735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.419834 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.420204 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.420207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.420209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.420473 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:33.420592 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.421217 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.421369 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:11:33.422492 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.407198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:39951 (local address 127.28.21.2:55843)
0806 02:11:33.407975 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.408035 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.408935 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.409607 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.409619 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.410216 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:33.411045 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.411064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.412396 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.412399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.413425 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.413435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.414261 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.414265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.414526 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.415238 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:33.415256 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:33.417177 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:33.419309 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.419315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.419331 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.419615 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.419933 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.419938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.419940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.420092 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.420642 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:33.420647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:33.420934 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.421128 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.421430 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":136,"threads_started":1}
I20260806 02:11:33.423875 15890 heartbeater.cc:347] Connected to a master server at 127.28.21.62:39951
I20260806 02:11:33.424170 15890 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:33.424723 15890 heartbeater.cc:511] Master 127.28.21.62:39951 requested a full tablet report, sending...
I20260806 02:11:33.425920 15551 ts_manager.cc:194] Registered new tserver with Master: 6699f7fc1dce40b8a09a119f01be5b06 (127.28.21.2:44919)
I20260806 02:11:33.426672 15551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:55843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:11:33.441717 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:33.552927 15895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:33.553165 15895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:33.553241 15895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:33.557242 15895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:33.557300 15895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:33.557403 15895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:33.562544 15895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:39951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:33.563521 15895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:33.564509 15895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:33.574051 15904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:33.574045 15901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:33.574062 15895 server_base.cc:1034] running on GCE node
W20260806 02:11:33.574016 15900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:33.574703 15895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:33.575284 15895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:33.576434 15895 hybrid_clock.cc:648] HybridClock initialized: now 1785982293576417 us; error 27 us; skew 500 ppm
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:33.579103 15895 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:11:33.580005 15895 webserver.cc:460] Webserver started at http://127.28.21.3:43351/ using document root <none> and password file <none>
I20260806 02:11:33.580499 15895 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:33.580555 15895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:33.580708 15895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:33.582809 15895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "8ee3b26a6fa84439a6793cea725441c0"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "f8376142e372906c44a885b011f1267e"
server_key_iv: "2d4fb5d60935c20803967422673098e7"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.583302 15895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "8ee3b26a6fa84439a6793cea725441c0"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "f8376142e372906c44a885b011f1267e"
server_key_iv: "2d4fb5d60935c20803967422673098e7"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.586906 15895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:11:33.589182 15912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.590611 15895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:33.590747 15895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "8ee3b26a6fa84439a6793cea725441c0"
format_stamp: "Formatted at 2026-08-06 02:11:33 on dist-test-slave-gp6n"
server_key: "f8376142e372906c44a885b011f1267e"
server_key_iv: "2d4fb5d60935c20803967422673098e7"
server_key_version: "encryptionkey@0"
I20260806 02:11:33.590864 15895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:33.616612 15895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:33.620512 15895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:33.620712 15895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:33.621321 15895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:33.622179 15895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:33.622226 15895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.622288 15895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:33.622320 15895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:33.631081 15895 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:43319
I20260806 02:11:33.631173 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:43319 every 8 connection(s)
I20260806 02:11:33.631996 15895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:33.638808 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 15895
I20260806 02:11:33.638949 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:33.639298 28756 external_mini_cluster.cc:1428] Setting key d21d4b68c958ba466e82af9a3bdb0c54
Aug 06 02:11:33 dist-test-slave-gp6n krb5kdc[15505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982293, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:33.645874 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.633881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:58027 (local address 127.28.21.62:39951)
0806 02:11:33.634050 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.634055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.635153 (+ 1098us) server_negotiation.cc:407] Connection header received
0806 02:11:33.635968 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.635971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.636018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.636094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:33.637240 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.637787 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.639312 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.639600 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.639693 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.642573 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:33.642576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:33.642578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:33.642603 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:33.644019 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.644541 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.644544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.644546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.644634 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.644942 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.644945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.644946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.645122 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:33.645201 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.645654 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.645754 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:11:33.646651 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.634316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:39951 (local address 127.28.21.3:58027)
0806 02:11:33.635006 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.635047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.635787 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.636244 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.636250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.636636 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:33.637105 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.637119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.637913 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.637916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.639188 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.639199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.639746 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.639754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.639957 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.640604 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:33.640619 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:33.642371 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:33.644128 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.644133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.644144 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.644412 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.644711 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:33.644713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:33.644715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:33.644850 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:33.645209 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:33.645214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:33.645440 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.645583 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.645823 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":114,"threads_started":1}
I20260806 02:11:33.647717 16032 heartbeater.cc:347] Connected to a master server at 127.28.21.62:39951
I20260806 02:11:33.647949 16032 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:33.648458 16032 heartbeater.cc:511] Master 127.28.21.62:39951 requested a full tablet report, sending...
I20260806 02:11:33.649425 15551 ts_manager.cc:194] Registered new tserver with Master: 8ee3b26a6fa84439a6793cea725441c0 (127.28.21.3:43319)
I20260806 02:11:33.649987 15551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:58027
I20260806 02:11:33.653532 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:33.663676 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.655571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.28.21.62:39951)
0806 02:11:33.655717 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.655722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.655932 (+ 210us) server_negotiation.cc:407] Connection header received
0806 02:11:33.656036 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.656038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.656078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.656136 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:33.656838 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.657330 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.658179 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.658498 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.658582 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.660306 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:33.660310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:33.660313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:33.660339 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:33.661673 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.662239 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.662242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.662243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.662324 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:33.662759 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:33.662761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:33.662762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:33.662985 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:33.663100 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.663370 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.663486 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:11:33.666854 15551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:33.669801 15551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:33.684558 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.678075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41677 (local address 127.0.0.1:50922)
0806 02:11:33.678542 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.678595 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.678732 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.679617 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.679624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.679650 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:33.679818 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.679826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681755 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.681759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.683163 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.683175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.683982 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.683986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.684099 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684147 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.684255 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.684373 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":81,"thread_start_us":81,"threads_started":1}
I20260806 02:11:33.684543 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.678218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.28.21.1:41677)
0806 02:11:33.679042 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.679057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.679081 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:11:33.679160 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.679167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.679369 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.679994 (+ 625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:33.680159 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681360 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.683308 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.683862 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.683981 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684135 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.684286 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.684390 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":84,"threads_started":1}
I20260806 02:11:33.684978 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.679210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:44919 (local address 127.0.0.1:58302)
0806 02:11:33.679535 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.679550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.679639 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.680398 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.680401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.680418 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:33.680501 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.680506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681595 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.681598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.682967 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.682977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.684212 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.684215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.684302 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684325 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.684387 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.684437 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":35,"thread_start_us":101,"threads_started":1}
I20260806 02:11:33.685048 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.678686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.28.21.3:43319)
0806 02:11:33.679156 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.679166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.679185 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:11:33.679246 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.679251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.679400 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.680071 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:33.680218 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681431 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.683287 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.684037 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.684518 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684776 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.684821 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.684912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":103,"threads_started":1}
I20260806 02:11:33.685348 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.679614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.28.21.2:44919)
0806 02:11:33.679919 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.679928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.679945 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:11:33.680077 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.680082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.680272 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.680418 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:33.680589 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681453 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.683088 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.683906 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.684014 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684162 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.684706 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.685176 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260806 02:11:33.684605 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.678086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:43319 (local address 127.0.0.1:38520)
0806 02:11:33.678698 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.678709 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.678780 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.680014 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.680017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.680037 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:33.680126 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.680131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.681534 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.681539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.683185 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.683191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.684165 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.684169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.684262 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.684298 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.684350 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.684437 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":87,"spinlock_wait_cycles":1920,"thread_start_us":107,"threads_started":1}
I20260806 02:11:33.687796 15822 tablet_service.cc:1467] Processing CreateTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c (DEFAULT_TABLE table=test-table [id=8d274697169845aabb98076e74312862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:33.687865 15683 tablet_service.cc:1467] Processing CreateTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c (DEFAULT_TABLE table=test-table [id=8d274697169845aabb98076e74312862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:33.688807 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145dd42fac3a4f6daac763e1eb20c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:33.688867 15683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145dd42fac3a4f6daac763e1eb20c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:33.688591 15962 tablet_service.cc:1467] Processing CreateTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c (DEFAULT_TABLE table=test-table [id=8d274697169845aabb98076e74312862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:33.689484 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145dd42fac3a4f6daac763e1eb20c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:33.694980 16049 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Bootstrap starting.
I20260806 02:11:33.696605 16050 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: Bootstrap starting.
I20260806 02:11:33.696605 16051 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Bootstrap starting.
I20260806 02:11:33.697832 16049 tablet_bootstrap.cc:654] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:33.698652 16049 log.cc:824] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:33.698789 16050 tablet_bootstrap.cc:654] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:33.698789 16051 tablet_bootstrap.cc:654] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:33.700102 16050 log.cc:824] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:33.700102 16051 log.cc:824] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:33.700798 16049 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: No bootstrap required, opened a new log
I20260806 02:11:33.701015 16049 ts_tablet_manager.cc:1397] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:33.702124 16051 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: No bootstrap required, opened a new log
I20260806 02:11:33.702121 16050 tablet_bootstrap.cc:492] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: No bootstrap required, opened a new log
I20260806 02:11:33.702296 16051 ts_tablet_manager.cc:1397] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:33.702296 16050 ts_tablet_manager.cc:1397] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:33.704008 16049 raft_consensus.cc:348] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.704253 16049 raft_consensus.cc:374] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:33.704306 16049 raft_consensus.cc:729] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6699f7fc1dce40b8a09a119f01be5b06, State: Initialized, Role: FOLLOWER
I20260806 02:11:33.704742 16049 consensus_queue.cc:260] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.705416 15890 heartbeater.cc:503] Master 127.28.21.62:39951 was elected leader, sending a full tablet report...
I20260806 02:11:33.705724 16049 ts_tablet_manager.cc:1428] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:11:33.706094 16050 raft_consensus.cc:348] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.706094 16051 raft_consensus.cc:348] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.706331 16051 raft_consensus.cc:374] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:33.706393 16051 raft_consensus.cc:729] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3b26a6fa84439a6793cea725441c0, State: Initialized, Role: FOLLOWER
I20260806 02:11:33.706332 16050 raft_consensus.cc:374] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:33.706499 16050 raft_consensus.cc:729] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdd9555711d49a188706d151da2647e, State: Initialized, Role: FOLLOWER
I20260806 02:11:33.706848 16051 consensus_queue.cc:260] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.707131 16050 consensus_queue.cc:260] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.707697 15752 heartbeater.cc:503] Master 127.28.21.62:39951 was elected leader, sending a full tablet report...
I20260806 02:11:33.707752 16051 ts_tablet_manager.cc:1428] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 02:11:33.708117 16032 heartbeater.cc:503] Master 127.28.21.62:39951 was elected leader, sending a full tablet report...
I20260806 02:11:33.709038 16050 ts_tablet_manager.cc:1428] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:11:33.859974 16057 raft_consensus.cc:482] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:33.860215 16057 raft_consensus.cc:504] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.861570 16057 leader_election.cc:288] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bdd9555711d49a188706d151da2647e (127.28.21.1:41677), 6699f7fc1dce40b8a09a119f01be5b06 (127.28.21.2:44919)
I20260806 02:11:33.866441 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.861861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41677 (local address 127.28.21.3:50185)
0806 02:11:33.862020 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.862038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.862146 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.862624 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.862627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.862643 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:33.862766 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.862771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.863847 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.863851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.865089 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.865099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.865949 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.865952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.866073 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.866093 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.866187 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.866267 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:11:33.866457 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.862144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:50185 (local address 127.28.21.1:41677)
0806 02:11:33.862276 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.862281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.862298 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:11:33.862448 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.862452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.862513 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.862613 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:33.862903 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.863713 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.865228 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.865809 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.865948 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.866019 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.866208 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.866286 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:11:33.867224 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.862058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:34871 (local address 127.28.21.2:44919)
0806 02:11:33.862199 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.862206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.862848 (+ 642us) server_negotiation.cc:407] Connection header received
0806 02:11:33.863012 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.863015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.863063 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.863149 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:33.863438 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.864097 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.865548 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.866185 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.866282 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.866341 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.867025 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.867106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:11:33.867201 15703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145dd42fac3a4f6daac763e1eb20c53c" candidate_uuid: "8ee3b26a6fa84439a6793cea725441c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdd9555711d49a188706d151da2647e" is_pre_election: true
I20260806 02:11:33.867578 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.861869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:44919 (local address 127.28.21.3:34871)
0806 02:11:33.862744 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.862758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:33.862887 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:33.863169 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:33.863172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:33.863189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:33.863321 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.863327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.864241 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.864244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.865406 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:33.865417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.866426 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.866430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:33.866852 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.866877 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:11:33.867371 (+ 494us) client_negotiation.cc:240] Negotiation successful
0806 02:11:33.867444 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":353,"thread_start_us":185,"threads_started":1}
I20260806 02:11:33.867585 15703 raft_consensus.cc:2454] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3b26a6fa84439a6793cea725441c0 in term 0.
I20260806 02:11:33.868181 15913 leader_election.cc:302] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bdd9555711d49a188706d151da2647e, 8ee3b26a6fa84439a6793cea725441c0; no voters:
I20260806 02:11:33.868184 15842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145dd42fac3a4f6daac763e1eb20c53c" candidate_uuid: "8ee3b26a6fa84439a6793cea725441c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6699f7fc1dce40b8a09a119f01be5b06" is_pre_election: true
I20260806 02:11:33.868459 15842 raft_consensus.cc:2454] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3b26a6fa84439a6793cea725441c0 in term 0.
I20260806 02:11:33.868484 16057 raft_consensus.cc:2781] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:33.868573 16057 raft_consensus.cc:482] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:33.868640 16057 raft_consensus.cc:3037] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:33.869800 16057 raft_consensus.cc:504] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.870221 16057 leader_election.cc:288] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [CANDIDATE]: Term 1 election: Requested vote from peers 0bdd9555711d49a188706d151da2647e (127.28.21.1:41677), 6699f7fc1dce40b8a09a119f01be5b06 (127.28.21.2:44919)
I20260806 02:11:33.870604 15842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145dd42fac3a4f6daac763e1eb20c53c" candidate_uuid: "8ee3b26a6fa84439a6793cea725441c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6699f7fc1dce40b8a09a119f01be5b06"
I20260806 02:11:33.870626 15703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145dd42fac3a4f6daac763e1eb20c53c" candidate_uuid: "8ee3b26a6fa84439a6793cea725441c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bdd9555711d49a188706d151da2647e"
I20260806 02:11:33.870743 15703 raft_consensus.cc:3037] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:33.870750 15842 raft_consensus.cc:3037] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:33.871919 15842 raft_consensus.cc:2454] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3b26a6fa84439a6793cea725441c0 in term 1.
I20260806 02:11:33.872102 15703 raft_consensus.cc:2454] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3b26a6fa84439a6793cea725441c0 in term 1.
I20260806 02:11:33.872273 15914 leader_election.cc:302] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6699f7fc1dce40b8a09a119f01be5b06, 8ee3b26a6fa84439a6793cea725441c0; no voters:
I20260806 02:11:33.872479 16057 raft_consensus.cc:2781] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:33.872716 16057 raft_consensus.cc:686] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3b26a6fa84439a6793cea725441c0, State: Running, Role: LEADER
I20260806 02:11:33.872993 16057 consensus_queue.cc:237] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } }
I20260806 02:11:33.876363 15551 catalog_manager.cc:5495] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3b26a6fa84439a6793cea725441c0 (127.28.21.3). New cstate: current_term: 1 leader_uuid: "8ee3b26a6fa84439a6793cea725441c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3b26a6fa84439a6793cea725441c0" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 } health_report { overall_health: UNKNOWN } } }
W20260806 02:11:33.882886 16033 tablet.cc:2369] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:11:33.905861 15891 tablet.cc:2369] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:33.918570 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.915255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.28.21.3:43319)
0806 02:11:33.915390 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:33.915393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:33.915409 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:11:33.915488 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:33.915490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:33.915548 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:33.915636 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:33.915852 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.916356 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.917169 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:33.917380 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:33.917464 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:33.917734 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:33.918131 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:33.918241 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:33.918355 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:11:33.918388 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:11:33.924305 15703 raft_consensus.cc:1261] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 1 FOLLOWER]: Refusing update from remote peer 8ee3b26a6fa84439a6793cea725441c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:33.924309 15842 raft_consensus.cc:1261] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3b26a6fa84439a6793cea725441c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:33.925036 16057 consensus_queue.cc:1035] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bdd9555711d49a188706d151da2647e" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:33.925261 16060 consensus_queue.cc:1035] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6699f7fc1dce40b8a09a119f01be5b06" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:33.932070 16063 mvcc.cc:205] Tried to move back new op lower bound from 7315383475906711552 to 7315383475706318848. Current Snapshot: MvccSnapshot[applied={T|T < 7315383475906711552}]
I20260806 02:11:33.933387 16064 mvcc.cc:205] Tried to move back new op lower bound from 7315383475906711552 to 7315383475706318848. Current Snapshot: MvccSnapshot[applied={T|T < 7315383475906711552}]
I20260806 02:11:33.937201 16066 mvcc.cc:205] Tried to move back new op lower bound from 7315383475906711552 to 7315383475706318848. Current Snapshot: MvccSnapshot[applied={T|T < 7315383475906711552 or (T in {7315383475906711552})}]
I20260806 02:11:33.940773 15549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:33.940919 15549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:33.943300 15549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4727a3b15e0f4440a9c301073ee04883: Sending DeleteTablet for 3 replicas of tablet 145dd42fac3a4f6daac763e1eb20c53c
I20260806 02:11:33.944062 15683 tablet_service.cc:1514] Processing DeleteTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:33 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:50922
I20260806 02:11:33.944160 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:33 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:38520
I20260806 02:11:33.944288 15822 tablet_service.cc:1514] Processing DeleteTablet for tablet 145dd42fac3a4f6daac763e1eb20c53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:33 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:58302
I20260806 02:11:33.944739 16073 tablet_replica.cc:330] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e: stopping tablet replica
I20260806 02:11:33.944974 16075 tablet_replica.cc:330] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: stopping tablet replica
I20260806 02:11:33.945246 16075 raft_consensus.cc:2229] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:33.945283 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15617
I20260806 02:11:33.945297 16073 raft_consensus.cc:2229] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:33.945554 16075 raft_consensus.cc:2258] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:33.945554 16073 raft_consensus.cc:2258] T 145dd42fac3a4f6daac763e1eb20c53c P 0bdd9555711d49a188706d151da2647e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:33.946178 16074 tablet_replica.cc:330] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: stopping tablet replica
I20260806 02:11:33.947212 16074 raft_consensus.cc:2229] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:11:33.947507 16075 ts_tablet_manager.cc:1905] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:33.947562 16074 raft_consensus.cc:2258] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:33.948985 16074 ts_tablet_manager.cc:1905] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:33.950274 16075 ts_tablet_manager.cc:1918] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:33.950368 16075 log.cc:1196] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/145dd42fac3a4f6daac763e1eb20c53c
I20260806 02:11:33.950730 16075 ts_tablet_manager.cc:1939] T 145dd42fac3a4f6daac763e1eb20c53c P 6699f7fc1dce40b8a09a119f01be5b06: Deleting consensus metadata
I20260806 02:11:33.951633 16074 ts_tablet_manager.cc:1918] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:33.951713 16074 log.cc:1196] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/145dd42fac3a4f6daac763e1eb20c53c
I20260806 02:11:33.951925 15536 catalog_manager.cc:4841] TS 6699f7fc1dce40b8a09a119f01be5b06 (127.28.21.2:44919): tablet 145dd42fac3a4f6daac763e1eb20c53c (table test-table [id=8d274697169845aabb98076e74312862]) successfully deleted
I20260806 02:11:33.952046 16074 ts_tablet_manager.cc:1939] T 145dd42fac3a4f6daac763e1eb20c53c P 8ee3b26a6fa84439a6793cea725441c0: Deleting consensus metadata
I20260806 02:11:33.952978 15536 catalog_manager.cc:4841] TS 8ee3b26a6fa84439a6793cea725441c0 (127.28.21.3:43319): tablet 145dd42fac3a4f6daac763e1eb20c53c (table test-table [id=8d274697169845aabb98076e74312862]) successfully deleted
W20260806 02:11:33.960727 15535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:41677 (error 108)
I20260806 02:11:33.960850 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15757
I20260806 02:11:33.962970 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:33.962628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41677 (local address 127.0.0.1:50930)
0806 02:11:33.962805 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:33.962870 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260806 02:11:33.963142 15535 catalog_manager.cc:4584] TS 0bdd9555711d49a188706d151da2647e (127.28.21.1:41677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 145dd42fac3a4f6daac763e1eb20c53c: Network error: Client connection negotiation failed: client connection to 127.28.21.1:41677: connect: Connection refused (error 111)
I20260806 02:11:33.968323 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15895
I20260806 02:11:33.975783 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 15518
2026-08-06T02:11:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2527 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:34Z Disabled control of system clock
I20260806 02:11:34.007264 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32799
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:32799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:34.121548 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:34.121812 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:34.121850 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:34.125972 16083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:34.126035 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:34.126051 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:34.126068 16083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:34.126083 16083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:34.130960 16083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:32799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:32799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:34.132063 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:34.133461 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:34.142095 16088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.142095 16089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.142351 16083 server_base.cc:1034] running on GCE node
W20260806 02:11:34.142620 16091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.143177 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:34.144490 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:34.145649 16083 hybrid_clock.cc:648] HybridClock initialized: now 1785982294145617 us; error 45 us; skew 500 ppm
I20260806 02:11:34.147547 16083 webserver.cc:460] Webserver started at http://127.28.21.62:46593/ using document root <none> and password file <none>
I20260806 02:11:34.148104 16083 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:34.148165 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:34.148398 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:34.150476 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "81dbfb2974354cf8ace00a973521b1ff"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "61089f063a91881637c1c95c7a8137c7"
server_key_iv: "254c0202fb027e8d785f625e5092ce95"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.151055 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "81dbfb2974354cf8ace00a973521b1ff"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "61089f063a91881637c1c95c7a8137c7"
server_key_iv: "254c0202fb027e8d785f625e5092ce95"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.154757 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:11:34.157145 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.158392 16083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:34.158569 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "81dbfb2974354cf8ace00a973521b1ff"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "61089f063a91881637c1c95c7a8137c7"
server_key_iv: "254c0202fb027e8d785f625e5092ce95"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.158680 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:34.174475 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:34.175220 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:34.175446 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:34.182392 16083 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:32799
I20260806 02:11:34.182417 16150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:32799 every 8 connection(s)
I20260806 02:11:34.183452 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:34.186553 16151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:34.192193 16151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: Bootstrap starting.
I20260806 02:11:34.193492 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16083
I20260806 02:11:34.193615 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:34.193922 28756 external_mini_cluster.cc:1428] Setting key 4b22b52c10bba23c1debe37650ab1ded
I20260806 02:11:34.195200 16151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:34.195941 16151 log.cc:824] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:34.198130 16151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: No bootstrap required, opened a new log
I20260806 02:11:34.202492 16151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } }
I20260806 02:11:34.202770 16151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:34.202845 16151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dbfb2974354cf8ace00a973521b1ff, State: Initialized, Role: FOLLOWER
I20260806 02:11:34.203348 16151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } }
I20260806 02:11:34.203510 16151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:34.203579 16151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:34.203679 16151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:34.203708 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.195148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.28.21.62:32799)
0806 02:11:34.195784 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.195798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.195835 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:11:34.196625 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.196649 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.197049 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.197329 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:34.197664 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.198765 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.200052 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.200568 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.200697 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.201679 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:34.201684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:34.201696 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:34.202081 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:34.202138 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:34.202158 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:34.202237 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.202496 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.202797 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":153,"threads_started":1}
I20260806 02:11:34.204820 16151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } }
I20260806 02:11:34.205174 16151 leader_election.cc:302] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81dbfb2974354cf8ace00a973521b1ff; no voters:
I20260806 02:11:34.205510 16151 leader_election.cc:288] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:34.205583 16157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:34.205776 16157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [term 1 LEADER]: Becoming Leader. State: Replica: 81dbfb2974354cf8ace00a973521b1ff, State: Running, Role: LEADER
I20260806 02:11:34.206084 16157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } }
I20260806 02:11:34.206817 16151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:34.207818 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81dbfb2974354cf8ace00a973521b1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } } }
I20260806 02:11:34.207844 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81dbfb2974354cf8ace00a973521b1ff. Latest consensus state: current_term: 1 leader_uuid: "81dbfb2974354cf8ace00a973521b1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dbfb2974354cf8ace00a973521b1ff" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 32799 } } }
I20260806 02:11:34.207952 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:34.207971 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:34.208335 16168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:34.211146 16168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:34.216890 16168 catalog_manager.cc:1347] Generated new cluster ID: 2b01c823f2d24f5b8a3bc545ccbe9fca
I20260806 02:11:34.216953 16168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:34.225401 16168 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:34.226677 16168 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:34.237807 16168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: Generated new TSK 0
I20260806 02:11:34.238655 16168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:34.241639 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--builtin_ntp_servers=127.28.21.20:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:34.363924 16178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:34.364200 16178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:34.364243 16178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:34.368919 16178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:34.368983 16178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:34.369046 16178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:34.373998 16178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:34.375116 16178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:34.376471 16178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:34.386060 16188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.386056 16185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.386222 16184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.386144 16178 server_base.cc:1034] running on GCE node
I20260806 02:11:34.386721 16178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:34.387341 16178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:34.388545 16178 hybrid_clock.cc:648] HybridClock initialized: now 1785982294388529 us; error 25 us; skew 500 ppm
I20260806 02:11:34.390704 16178 webserver.cc:460] Webserver started at http://127.28.21.1:39779/ using document root <none> and password file <none>
I20260806 02:11:34.391337 16178 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:34.391415 16178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:34.391644 16178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:34.393882 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "51783ef9fa3197cee2ff9aeb93a81f48"
server_key_iv: "145f891660d364eeb130046c0bd666b9"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.394618 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "51783ef9fa3197cee2ff9aeb93a81f48"
server_key_iv: "145f891660d364eeb130046c0bd666b9"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.398804 16178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:11:34.401237 16194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.402567 16178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:34.402691 16178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "51783ef9fa3197cee2ff9aeb93a81f48"
server_key_iv: "145f891660d364eeb130046c0bd666b9"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.402799 16178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:34.416338 16178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:34.416991 16178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:34.417198 16178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:34.417763 16178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:34.418743 16178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:34.418812 16178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.418874 16178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:34.418919 16178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.428031 16178 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:41005
I20260806 02:11:34.428059 16311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:41005 every 8 connection(s)
I20260806 02:11:34.428958 16178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:34.438980 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16178
I20260806 02:11:34.438987 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.430946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:37705 (local address 127.28.21.62:32799)
0806 02:11:34.431089 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.431093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.432156 (+ 1063us) server_negotiation.cc:407] Connection header received
0806 02:11:34.433799 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.433802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.433867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.433976 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:34.435006 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.435606 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.436862 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.437125 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.437248 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.438447 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:34.438452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:34.438455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:34.438473 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:34.438492 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:34.438500 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:34.438581 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.438762 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.438828 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:11:34.439153 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:34.439499 28756 external_mini_cluster.cc:1428] Setting key 7b5214d3d01bbde4c8d5b0c1b9823562
I20260806 02:11:34.440075 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.431308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32799 (local address 127.28.21.1:37705)
0806 02:11:34.431996 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.432041 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.433553 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.434162 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.434170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.434258 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:34.434838 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.434860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.435750 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.435759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.436725 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.436735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.437289 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.437293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.437471 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.438138 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:34.438140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:34.438181 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:34.438246 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:34.438597 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:34.438609 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:34.438621 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.438758 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.439061 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260806 02:11:34.441358 16312 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32799
I20260806 02:11:34.441581 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--builtin_ntp_servers=127.28.21.20:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:34.441604 16312 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:34.442181 16312 heartbeater.cc:511] Master 127.28.21.62:32799 requested a full tablet report, sending...
I20260806 02:11:34.443791 16114 ts_manager.cc:194] Registered new tserver with Master: a1cbc279d4c84c15a00a5ffe5e8b504f (127.28.21.1:41005)
I20260806 02:11:34.445039 16114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.1:37705
W20260806 02:11:34.554065 16315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:34.554378 16315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:34.554471 16315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:34.558530 16315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:34.558596 16315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:34.558678 16315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:34.563573 16315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:34.564683 16315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:34.566102 16315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:11:34.576032 16315 server_base.cc:1034] running on GCE node
W20260806 02:11:34.575945 16320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.575939 16321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.576323 16323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.576752 16315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:34.577343 16315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:34.578488 16315 hybrid_clock.cc:648] HybridClock initialized: now 1785982294578454 us; error 42 us; skew 500 ppm
I20260806 02:11:34.580403 16315 webserver.cc:460] Webserver started at http://127.28.21.2:36557/ using document root <none> and password file <none>
I20260806 02:11:34.580976 16315 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:34.581041 16315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:34.581248 16315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:34.583444 16315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "3827ffe3d7864ef1be2b67f2f932c5ab"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "d9d532ec4fb5af746dde0dc5dea299de"
server_key_iv: "f0f23fa0763ad264ed5f2c08fe17dcbd"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.584008 16315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "3827ffe3d7864ef1be2b67f2f932c5ab"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "d9d532ec4fb5af746dde0dc5dea299de"
server_key_iv: "f0f23fa0763ad264ed5f2c08fe17dcbd"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.587680 16315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:34.590101 16330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.591365 16315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:34.591519 16315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "3827ffe3d7864ef1be2b67f2f932c5ab"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "d9d532ec4fb5af746dde0dc5dea299de"
server_key_iv: "f0f23fa0763ad264ed5f2c08fe17dcbd"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.591631 16315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:34.622900 16315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:34.623610 16315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:34.623812 16315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:34.624440 16315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:34.625391 16315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:34.625461 16315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.625540 16315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:34.625586 16315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.634874 16315 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:36029
I20260806 02:11:34.634912 16447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:36029 every 8 connection(s)
I20260806 02:11:34.635803 16315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:34.638753 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16315
I20260806 02:11:34.638895 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:34.639213 28756 external_mini_cluster.cc:1428] Setting key f3ff18c6659f855e47f427eff488b3f4
I20260806 02:11:34.641312 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--builtin_ntp_servers=127.28.21.20:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:34.646449 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.637774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:39497 (local address 127.28.21.62:32799)
0806 02:11:34.637926 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.637930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.639104 (+ 1174us) server_negotiation.cc:407] Connection header received
0806 02:11:34.640888 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.640892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.640950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.641062 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:34.642131 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.642803 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.643999 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.644450 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.644590 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.645702 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:34.645710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:34.645717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:34.645737 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:34.645767 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:34.645782 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:34.645878 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.646073 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.646158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:11:34.647156 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.638127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32799 (local address 127.28.21.2:39497)
0806 02:11:34.638908 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.638975 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.640642 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.641257 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.641265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.641358 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:34.641973 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.641988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.642961 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.642967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.643873 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.643882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.644600 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.644602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.644751 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.645426 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:34.645428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:34.645469 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:34.645534 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:34.645904 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:34.645920 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:34.645926 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.646039 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.646264 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":118,"threads_started":1}
I20260806 02:11:34.648345 16448 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32799
I20260806 02:11:34.648609 16448 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:34.649098 16448 heartbeater.cc:511] Master 127.28.21.62:32799 requested a full tablet report, sending...
I20260806 02:11:34.650126 16114 ts_manager.cc:194] Registered new tserver with Master: 3827ffe3d7864ef1be2b67f2f932c5ab (127.28.21.2:36029)
I20260806 02:11:34.650753 16114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.2:39497
W20260806 02:11:34.758049 16451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:34.758301 16451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:34.758369 16451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:34.762473 16451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:34.762526 16451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:34.762602 16451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:34.767382 16451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:34.768435 16451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:34.769923 16451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:34.779881 16460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:34.779932 16457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.779991 16451 server_base.cc:1034] running on GCE node
W20260806 02:11:34.779899 16458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:34.780663 16451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:34.781296 16451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:34.782488 16451 hybrid_clock.cc:648] HybridClock initialized: now 1785982294782439 us; error 55 us; skew 500 ppm
I20260806 02:11:34.784271 16451 webserver.cc:460] Webserver started at http://127.28.21.3:45073/ using document root <none> and password file <none>
I20260806 02:11:34.784808 16451 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:34.784893 16451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:34.785064 16451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:34.787200 16451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "f006ae65f65144e6b630894becdd557a"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "6e7496a15e5601304db61cd20d77767d"
server_key_iv: "0367374617ffa1ff33fa5f1bd5c9de30"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.787710 16451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "f006ae65f65144e6b630894becdd557a"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "6e7496a15e5601304db61cd20d77767d"
server_key_iv: "0367374617ffa1ff33fa5f1bd5c9de30"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.791415 16451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:34.793670 16466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.795044 16451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:34.795171 16451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "f006ae65f65144e6b630894becdd557a"
format_stamp: "Formatted at 2026-08-06 02:11:34 on dist-test-slave-gp6n"
server_key: "6e7496a15e5601304db61cd20d77767d"
server_key_iv: "0367374617ffa1ff33fa5f1bd5c9de30"
server_key_version: "encryptionkey@0"
I20260806 02:11:34.795274 16451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:34.807778 16451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:34.808471 16451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:34.808653 16451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:34.809198 16451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:34.810112 16451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:34.810159 16451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.810223 16451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:34.810259 16451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:34.818893 16451 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:41103
I20260806 02:11:34.818933 16584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:41103 every 8 connection(s)
I20260806 02:11:34.819779 16451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:34.828627 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16451
I20260806 02:11:34.828773 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:34.828981 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.821616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:60139 (local address 127.28.21.62:32799)
0806 02:11:34.821760 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.821764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.822845 (+ 1081us) server_negotiation.cc:407] Connection header received
0806 02:11:34.824238 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.824241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.824297 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.824422 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:34.825307 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.825845 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.827030 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.827265 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.827389 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.828489 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:34.828492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:34.828494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:34.828507 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:34.828525 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:34.828533 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:34.828606 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.828769 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.828834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:11:34.829126 28756 external_mini_cluster.cc:1428] Setting key 445ebc8b747c2b1a679c36f8275d5c57
I20260806 02:11:34.830032 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.822021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:32799 (local address 127.28.21.3:60139)
0806 02:11:34.822700 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.822740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.824030 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.824588 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.824595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.824679 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:34.825162 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.825174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.825972 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.825976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.826902 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.826912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.827421 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.827425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.827597 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.828218 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:34.828220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:34.828260 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:34.828323 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:34.828620 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:34.828631 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:34.828642 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.828762 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.829055 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260806 02:11:34.831141 16585 heartbeater.cc:347] Connected to a master server at 127.28.21.62:32799
I20260806 02:11:34.831363 16585 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:34.831857 16585 heartbeater.cc:511] Master 127.28.21.62:32799 requested a full tablet report, sending...
I20260806 02:11:34.832824 16114 ts_manager.cc:194] Registered new tserver with Master: f006ae65f65144e6b630894becdd557a (127.28.21.3:41103)
I20260806 02:11:34.833384 16114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.3:60139
I20260806 02:11:34.842938 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:34.848882 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.844880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.28.21.62:32799)
0806 02:11:34.845036 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.845041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.845216 (+ 175us) server_negotiation.cc:407] Connection header received
0806 02:11:34.845304 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.845306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.845346 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.845449 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:34.845658 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.846161 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.847137 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.847414 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.847508 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.848324 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:34.848327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:34.848329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:34.848375 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:34.848394 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:34.848402 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:34.848506 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.848655 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.848705 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:11:34.851992 16114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:34.855130 16114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:34.869436 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.864122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.28.21.1:41005)
0806 02:11:34.864555 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.864566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.864585 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:11:34.864648 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.864652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.864804 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.865354 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:34.865493 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.866638 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868030 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.868741 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868864 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.869056 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.869172 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.869273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":121,"threads_started":1}
I20260806 02:11:34.869445 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.864034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36029 (local address 127.0.0.1:57344)
0806 02:11:34.864649 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.864665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.864746 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.865751 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.865754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.865772 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:34.865857 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.865862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.867079 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.867086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.868016 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.868026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.869082 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.869086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.869177 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.869197 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.869260 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.869320 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":65,"thread_start_us":89,"threads_started":1}
I20260806 02:11:34.869445 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.863637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41005 (local address 127.0.0.1:46150)
0806 02:11:34.864196 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.864211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.864304 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.864960 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.864963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.864991 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:34.865191 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.865200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.866753 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.866756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.867907 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.867915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.868880 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.868991 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.869034 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.869143 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.869271 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":33,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260806 02:11:34.870127 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.863290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:41103 (local address 127.0.0.1:60454)
0806 02:11:34.863671 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:34.863724 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:34.863858 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:34.864949 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:34.864954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:34.864989 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:34.865224 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.865234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.866344 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.866350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.867766 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:34.867776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.869498 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.869502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:34.869819 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.869843 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:11:34.869927 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:11:34.869995 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":67,"spinlock_wait_cycles":2944,"thread_start_us":105,"threads_started":1}
I20260806 02:11:34.870229 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.863448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.28.21.3:41103)
0806 02:11:34.864441 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.864455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.864481 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:11:34.864560 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.864566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.864781 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.864954 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:34.865342 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.866240 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.867912 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.868741 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.869263 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.869982 (+ 719us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.870082 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":126,"threads_started":1}
I20260806 02:11:34.870657 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:34.864229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.28.21.2:36029)
0806 02:11:34.865293 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:34.865306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:34.865329 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:11:34.865417 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:34.865421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:34.865603 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:34.865754 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:34.865976 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.866974 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868149 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:34.868791 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:34.868912 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:34.870110 (+ 1198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:34.870162 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:11:34.870541 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":153,"threads_started":1}
I20260806 02:11:34.872865 16381 tablet_service.cc:1467] Processing CreateTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af (DEFAULT_TABLE table=test-table [id=2ee5732a54a845c3a05a28a5bf39201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:34.872856 16244 tablet_service.cc:1467] Processing CreateTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af (DEFAULT_TABLE table=test-table [id=2ee5732a54a845c3a05a28a5bf39201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:34.872891 16517 tablet_service.cc:1467] Processing CreateTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af (DEFAULT_TABLE table=test-table [id=2ee5732a54a845c3a05a28a5bf39201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:34.873832 16381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c2739cf8a54b97b1b39f0af80ff7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:34.873878 16244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c2739cf8a54b97b1b39f0af80ff7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:34.873878 16517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c2739cf8a54b97b1b39f0af80ff7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:34.880534 16603 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Bootstrap starting.
I20260806 02:11:34.881076 16604 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Bootstrap starting.
I20260806 02:11:34.882725 16605 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Bootstrap starting.
I20260806 02:11:34.883021 16604 tablet_bootstrap.cc:654] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:34.883021 16603 tablet_bootstrap.cc:654] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:34.883911 16603 log.cc:824] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:34.883910 16604 log.cc:824] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:34.884599 16605 tablet_bootstrap.cc:654] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:34.885408 16605 log.cc:824] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:34.885952 16603 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: No bootstrap required, opened a new log
I20260806 02:11:34.885952 16604 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: No bootstrap required, opened a new log
I20260806 02:11:34.886137 16604 ts_tablet_manager.cc:1397] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:11:34.886137 16603 ts_tablet_manager.cc:1397] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:11:34.887195 16605 tablet_bootstrap.cc:492] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: No bootstrap required, opened a new log
I20260806 02:11:34.887380 16605 ts_tablet_manager.cc:1397] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:11:34.890348 16605 raft_consensus.cc:348] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.890465 16604 raft_consensus.cc:348] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.890461 16603 raft_consensus.cc:348] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.890662 16605 raft_consensus.cc:374] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:34.890702 16603 raft_consensus.cc:374] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:34.890702 16604 raft_consensus.cc:374] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:34.890733 16605 raft_consensus.cc:729] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ae65f65144e6b630894becdd557a, State: Initialized, Role: FOLLOWER
I20260806 02:11:34.890758 16603 raft_consensus.cc:729] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3827ffe3d7864ef1be2b67f2f932c5ab, State: Initialized, Role: FOLLOWER
I20260806 02:11:34.890758 16604 raft_consensus.cc:729] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cbc279d4c84c15a00a5ffe5e8b504f, State: Initialized, Role: FOLLOWER
I20260806 02:11:34.891191 16605 consensus_queue.cc:260] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.891191 16603 consensus_queue.cc:260] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.891247 16604 consensus_queue.cc:260] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:34.891811 16585 heartbeater.cc:503] Master 127.28.21.62:32799 was elected leader, sending a full tablet report...
I20260806 02:11:34.892177 16603 ts_tablet_manager.cc:1428] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:11:34.892207 16312 heartbeater.cc:503] Master 127.28.21.62:32799 was elected leader, sending a full tablet report...
I20260806 02:11:34.892616 16604 ts_tablet_manager.cc:1428] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:11:34.892828 16448 heartbeater.cc:503] Master 127.28.21.62:32799 was elected leader, sending a full tablet report...
I20260806 02:11:34.893646 16605 ts_tablet_manager.cc:1428] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 02:11:34.930099 16313 tablet.cc:2369] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:35.008020 16609 raft_consensus.cc:482] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:35.008272 16609 raft_consensus.cc:504] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:35.009562 16609 leader_election.cc:288] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3827ffe3d7864ef1be2b67f2f932c5ab (127.28.21.2:36029), a1cbc279d4c84c15a00a5ffe5e8b504f (127.28.21.1:41005)
I20260806 02:11:35.013322 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.009877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41005 (local address 127.28.21.3:35697)
0806 02:11:35.010034 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:35.010053 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:35.010167 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:35.010506 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:35.010508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:35.010528 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:35.010658 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:35.010664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.011403 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.011407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:35.012205 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:35.012214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.012887 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.012890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:35.012989 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:35.013010 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:11:35.013105 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:11:35.013197 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:11:35.013322 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.010078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:35697 (local address 127.28.21.1:41005)
0806 02:11:35.010212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:35.010218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:35.010235 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:11:35.010299 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:35.010301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:35.010376 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:35.010491 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:35.010779 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.011280 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.012334 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.012752 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.012878 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:35.012939 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:35.013121 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:11:35.013188 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:11:35.014020 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.010001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:48917 (local address 127.28.21.2:36029)
0806 02:11:35.010161 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:35.010166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:35.010609 (+ 443us) server_negotiation.cc:407] Connection header received
0806 02:11:35.010759 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:35.010762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:35.010812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:35.010911 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:35.011181 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.011669 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.012697 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.013285 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.013394 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:35.013459 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:35.013797 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:11:35.013882 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:11:35.014194 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c2739cf8a54b97b1b39f0af80ff7af" candidate_uuid: "f006ae65f65144e6b630894becdd557a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" is_pre_election: true
I20260806 02:11:35.014451 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.009877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:36029 (local address 127.28.21.3:48917)
0806 02:11:35.010514 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:35.010527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:35.010650 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:35.010926 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:35.010928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:35.010944 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:35.011072 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:35.011077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.011784 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.011787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:35.012571 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:35.012580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.013549 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.013553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:35.013644 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:35.013675 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:11:35.014168 (+ 493us) client_negotiation.cc:240] Negotiation successful
0806 02:11:35.014244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":253,"thread_start_us":136,"threads_started":1}
I20260806 02:11:35.014604 16265 raft_consensus.cc:2454] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ae65f65144e6b630894becdd557a in term 0.
I20260806 02:11:35.015048 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c2739cf8a54b97b1b39f0af80ff7af" candidate_uuid: "f006ae65f65144e6b630894becdd557a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" is_pre_election: true
I20260806 02:11:35.015187 16469 leader_election.cc:302] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1cbc279d4c84c15a00a5ffe5e8b504f, f006ae65f65144e6b630894becdd557a; no voters:
I20260806 02:11:35.015360 16402 raft_consensus.cc:2454] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ae65f65144e6b630894becdd557a in term 0.
I20260806 02:11:35.015515 16609 raft_consensus.cc:2781] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:35.015630 16609 raft_consensus.cc:482] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:35.015699 16609 raft_consensus.cc:3037] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:35.016834 16609 raft_consensus.cc:504] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:35.017200 16609 leader_election.cc:288] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [CANDIDATE]: Term 1 election: Requested vote from peers 3827ffe3d7864ef1be2b67f2f932c5ab (127.28.21.2:36029), a1cbc279d4c84c15a00a5ffe5e8b504f (127.28.21.1:41005)
I20260806 02:11:35.017581 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c2739cf8a54b97b1b39f0af80ff7af" candidate_uuid: "f006ae65f65144e6b630894becdd557a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f"
I20260806 02:11:35.017725 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c2739cf8a54b97b1b39f0af80ff7af" candidate_uuid: "f006ae65f65144e6b630894becdd557a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab"
I20260806 02:11:35.017760 16265 raft_consensus.cc:3037] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:35.017871 16402 raft_consensus.cc:3037] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:35.019006 16265 raft_consensus.cc:2454] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ae65f65144e6b630894becdd557a in term 1.
I20260806 02:11:35.019045 16402 raft_consensus.cc:2454] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ae65f65144e6b630894becdd557a in term 1.
I20260806 02:11:35.019335 16469 leader_election.cc:302] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1cbc279d4c84c15a00a5ffe5e8b504f, f006ae65f65144e6b630894becdd557a; no voters:
I20260806 02:11:35.019533 16609 raft_consensus.cc:2781] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:35.019788 16609 raft_consensus.cc:686] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 1 LEADER]: Becoming Leader. State: Replica: f006ae65f65144e6b630894becdd557a, State: Running, Role: LEADER
I20260806 02:11:35.020126 16609 consensus_queue.cc:237] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } }
I20260806 02:11:35.023645 16115 catalog_manager.cc:5495] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a reported cstate change: term changed from 0 to 1, leader changed from <none> to f006ae65f65144e6b630894becdd557a (127.28.21.3). New cstate: current_term: 1 leader_uuid: "f006ae65f65144e6b630894becdd557a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f006ae65f65144e6b630894becdd557a" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 41103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20260806 02:11:35.059813 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.056066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.28.21.3:41103)
0806 02:11:35.056201 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:35.056206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:35.056222 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:11:35.056360 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:35.056362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:35.056422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:35.056518 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:35.056742 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.057281 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.058153 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:35.058427 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:35.058517 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:35.058753 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:35.059386 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:35.059518 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:35.059599 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 02:11:35.059660 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:35.066488 16265 raft_consensus.cc:1261] T 39c2739cf8a54b97b1b39f0af80ff7af P a1cbc279d4c84c15a00a5ffe5e8b504f [term 1 FOLLOWER]: Refusing update from remote peer f006ae65f65144e6b630894becdd557a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:35.066486 16402 raft_consensus.cc:1261] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 1 FOLLOWER]: Refusing update from remote peer f006ae65f65144e6b630894becdd557a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:35.067210 16609 consensus_queue.cc:1035] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1cbc279d4c84c15a00a5ffe5e8b504f" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:35.067437 16613 consensus_queue.cc:1035] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3827ffe3d7864ef1be2b67f2f932c5ab" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 36029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:11:35.070675 16586 tablet.cc:2369] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:35.076725 16619 mvcc.cc:205] Tried to move back new op lower bound from 7315383480584613888 to 7315383480405393408. Current Snapshot: MvccSnapshot[applied={T|T < 7315383480584613888}]
I20260806 02:11:35.078091 16618 mvcc.cc:205] Tried to move back new op lower bound from 7315383480584613888 to 7315383480405393408. Current Snapshot: MvccSnapshot[applied={T|T < 7315383480584613888}]
I20260806 02:11:35.083005 16115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:35.083140 16115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:35.085254 16115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81dbfb2974354cf8ace00a973521b1ff: Sending DeleteTablet for 3 replicas of tablet 39c2739cf8a54b97b1b39f0af80ff7af
I20260806 02:11:35.086203 16517 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:35 UTC) from {username='slave'} at 127.0.0.1:60454
I20260806 02:11:35.086193 16381 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:35 UTC) from {username='slave'} at 127.0.0.1:57344
I20260806 02:11:35.086715 16627 tablet_replica.cc:330] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: stopping tablet replica
I20260806 02:11:35.086720 16244 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c2739cf8a54b97b1b39f0af80ff7af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:35 UTC) from {username='slave'} at 127.0.0.1:46150
I20260806 02:11:35.087018 16628 tablet_replica.cc:330] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: stopping tablet replica
I20260806 02:11:35.087113 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16178
I20260806 02:11:35.087387 16627 raft_consensus.cc:2229] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:11:35.087724 16627 raft_consensus.cc:2258] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:35.088136 16628 raft_consensus.cc:2229] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:35.088423 16628 raft_consensus.cc:2258] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:35.090458 16628 ts_tablet_manager.cc:1905] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:35.090794 16627 ts_tablet_manager.cc:1905] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:35.093313 16628 ts_tablet_manager.cc:1918] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:35.093423 16628 log.cc:1196] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/39c2739cf8a54b97b1b39f0af80ff7af
I20260806 02:11:35.093569 16627 ts_tablet_manager.cc:1918] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:35.093725 16627 log.cc:1196] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/39c2739cf8a54b97b1b39f0af80ff7af
I20260806 02:11:35.093766 16628 ts_tablet_manager.cc:1939] T 39c2739cf8a54b97b1b39f0af80ff7af P 3827ffe3d7864ef1be2b67f2f932c5ab: Deleting consensus metadata
I20260806 02:11:35.094107 16627 ts_tablet_manager.cc:1939] T 39c2739cf8a54b97b1b39f0af80ff7af P f006ae65f65144e6b630894becdd557a: Deleting consensus metadata
I20260806 02:11:35.095095 16098 catalog_manager.cc:4841] TS 3827ffe3d7864ef1be2b67f2f932c5ab (127.28.21.2:36029): tablet 39c2739cf8a54b97b1b39f0af80ff7af (table test-table [id=2ee5732a54a845c3a05a28a5bf39201f]) successfully deleted
I20260806 02:11:35.095589 16098 catalog_manager.cc:4841] TS f006ae65f65144e6b630894becdd557a (127.28.21.3:41103): tablet 39c2739cf8a54b97b1b39f0af80ff7af (table test-table [id=2ee5732a54a845c3a05a28a5bf39201f]) successfully deleted
W20260806 02:11:35.102488 16100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:41005 (error 108)
I20260806 02:11:35.102629 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16315
I20260806 02:11:35.105432 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:35.105071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:41005 (local address 127.0.0.1:46166)
0806 02:11:35.105240 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:35.105327 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:41005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:11:35.105681 16100 catalog_manager.cc:4584] TS a1cbc279d4c84c15a00a5ffe5e8b504f (127.28.21.1:41005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39c2739cf8a54b97b1b39f0af80ff7af: Network error: Client connection negotiation failed: client connection to 127.28.21.1:41005: connect: Connection refused (error 111)
I20260806 02:11:35.109444 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16451
I20260806 02:11:35.115155 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16083
2026-08-06T02:11:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1139 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:35 dist-test-slave-gp6n krb5kdc[16633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:35 dist-test-slave-gp6n krb5kdc[16633](info): set up 2 sockets
Aug 06 02:11:35 dist-test-slave-gp6n krb5kdc[16633](info): commencing operation
krb5kdc: starting...
W20260806 02:11:36.211285 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:36 dist-test-slave-gp6n krb5kdc[16633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:11:36.344033 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36327
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:36327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:36.455808 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:36.456063 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:36.456102 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:36.460065 16646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:36.460120 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:36.460139 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:36.460156 16646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:36.460171 16646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:36.465293 16646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:36327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:36.466301 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:36.467329 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:36.476352 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:36.476347 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:36.476420 16646 server_base.cc:1034] running on GCE node
W20260806 02:11:36.476413 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:36.477221 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:36.478384 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:36.479630 16646 hybrid_clock.cc:648] HybridClock initialized: now 1785982296479605 us; error 47 us; skew 500 ppm
Aug 06 02:11:36 dist-test-slave-gp6n krb5kdc[16633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:36.486518 16646 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:36.487520 16646 webserver.cc:460] Webserver started at http://127.28.21.62:34221/ using document root <none> and password file <none>
I20260806 02:11:36.488111 16646 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:36.488204 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:36.488445 16646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:36.490651 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "9e42a937a9ca441ca08ddc5a97e3cefd"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "2a8475f6511a28f3ef7bff5a512e8140"
server_key_iv: "188ec920bc5a255128b212a0bc123de3"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.491240 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "9e42a937a9ca441ca08ddc5a97e3cefd"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "2a8475f6511a28f3ef7bff5a512e8140"
server_key_iv: "188ec920bc5a255128b212a0bc123de3"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.495086 16646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:11:36.497529 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:36.498987 16646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:36.499145 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "9e42a937a9ca441ca08ddc5a97e3cefd"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "2a8475f6511a28f3ef7bff5a512e8140"
server_key_iv: "188ec920bc5a255128b212a0bc123de3"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.499269 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:36.521214 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:36.525048 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:36.525274 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:36.533371 16646 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:36327
I20260806 02:11:36.533411 16714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:36327 every 8 connection(s)
I20260806 02:11:36.534518 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:36.537380 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:36.541167 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16646
I20260806 02:11:36.541328 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:36.541671 28756 external_mini_cluster.cc:1428] Setting key 00ae5fdc7b3002d9c551d5707b04ab6a
I20260806 02:11:36.543798 16715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: Bootstrap starting.
I20260806 02:11:36.546679 16715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:36.547617 16715 log.cc:824] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:36.549901 16715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: No bootstrap required, opened a new log
Aug 06 02:11:36 dist-test-slave-gp6n krb5kdc[16633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:36.554980 16715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } }
I20260806 02:11:36.555282 16715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:36.555351 16715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e42a937a9ca441ca08ddc5a97e3cefd, State: Initialized, Role: FOLLOWER
I20260806 02:11:36.555814 16715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } }
I20260806 02:11:36.555999 16715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:36.556099 16715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:36.556209 16715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:36.557447 16715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } }
I20260806 02:11:36.557823 16715 leader_election.cc:302] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e42a937a9ca441ca08ddc5a97e3cefd; no voters:
I20260806 02:11:36.558216 16715 leader_election.cc:288] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:36.558343 16721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:36.558643 16721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [term 1 LEADER]: Becoming Leader. State: Replica: 9e42a937a9ca441ca08ddc5a97e3cefd, State: Running, Role: LEADER
I20260806 02:11:36.558947 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:36.543257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.28.21.62:36327)
0806 02:11:36.543707 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:36.543732 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:36.543778 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:11:36.544653 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:36.544678 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:36.545133 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:36.545552 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:36.546308 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.548067 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.549097 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.549901 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.550091 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:36.553078 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:36.553087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:36.553112 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:36.553155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:36.555399 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:36.555968 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:36.555974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:36.555980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:36.556169 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:36.556511 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:36.556514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:36.556516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:36.557042 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:36.557270 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:36.557506 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:11:36.557823 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":117,"threads_started":1}
I20260806 02:11:36.558991 16721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } }
I20260806 02:11:36.559559 16715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:36.560910 16722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e42a937a9ca441ca08ddc5a97e3cefd. Latest consensus state: current_term: 1 leader_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } } }
I20260806 02:11:36.560961 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e42a937a9ca441ca08ddc5a97e3cefd" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36327 } } }
I20260806 02:11:36.561039 16722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:36.561064 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:36.562093 16731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:36.565896 16731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:36.571857 16731 catalog_manager.cc:1347] Generated new cluster ID: 207d9c2c0d3a487db207ee4cdf476ba7
I20260806 02:11:36.572000 16731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:36.588266 16731 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:36.589984 16731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:36.597481 16731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: Generated new TSK 0
I20260806 02:11:36.598222 16731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
Entry for principal kudu/127.28.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab.
I20260806 02:11:36.635506 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:36.770989 16744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:36.771234 16744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:36.771277 16744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:36.775413 16744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:36.775475 16744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:36.775532 16744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:36.780804 16744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.1.keytab
--principal=kudu/127.28.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:36.781725 16744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:36.782775 16744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:36.792076 16749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:36.792089 16750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:36.792098 16752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:36.792493 16744 server_base.cc:1034] running on GCE node
I20260806 02:11:36.792901 16744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:36.793601 16744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:36.794832 16744 hybrid_clock.cc:648] HybridClock initialized: now 1785982296794796 us; error 47 us; skew 500 ppm
Aug 06 02:11:36 dist-test-slave-gp6n krb5kdc[16633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:36.797849 16744 init.cc:377] Logged in from keytab as kudu/127.28.21.1@KRBTEST.COM (short username kudu)
I20260806 02:11:36.798869 16744 webserver.cc:460] Webserver started at http://127.28.21.1:34717/ using document root <none> and password file <none>
I20260806 02:11:36.799485 16744 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:36.799587 16744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:36.799834 16744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:36.802006 16744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "fb74c5df21c34443b06b2a74ee426394"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "738a5c0509a3fc258c3696fb99be3076"
server_key_iv: "c73be7eadba615dab1cc966793fbad96"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.802752 16744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "fb74c5df21c34443b06b2a74ee426394"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "738a5c0509a3fc258c3696fb99be3076"
server_key_iv: "c73be7eadba615dab1cc966793fbad96"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.806979 16744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:11:36.809571 16759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:36.810962 16744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:36.811103 16744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "fb74c5df21c34443b06b2a74ee426394"
format_stamp: "Formatted at 2026-08-06 02:11:36 on dist-test-slave-gp6n"
server_key: "738a5c0509a3fc258c3696fb99be3076"
server_key_iv: "c73be7eadba615dab1cc966793fbad96"
server_key_version: "encryptionkey@0"
I20260806 02:11:36.811214 16744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:36.822844 16744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:36.825873 16744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:36.826094 16744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:36.826764 16744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:36.827633 16744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:36.827713 16744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:36.827808 16744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:36.827883 16744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:36.837672 16744 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:45055
I20260806 02:11:36.837711 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:45055 every 8 connection(s)
I20260806 02:11:36.838629 16744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:36.843405 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16744
I20260806 02:11:36.843555 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:36.843942 28756 external_mini_cluster.cc:1428] Setting key 59a0762f2389d60fa61cbcd1b3941a5c
Aug 06 02:11:36 dist-test-slave-gp6n krb5kdc[16633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.1@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.2@KRBTEST.COM; defaulting to no policy
I20260806 02:11:36.855010 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:36.840506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:52789 (local address 127.28.21.62:36327)
0806 02:11:36.840669 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:36.840674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:36.841643 (+ 969us) server_negotiation.cc:407] Connection header received
0806 02:11:36.843179 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:36.843182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:36.843229 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:36.843311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:36.844992 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.845732 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.847705 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.848117 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.848236 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:36.851079 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:36.851083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:36.851085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:36.851109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:36.852663 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:36.853235 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:36.853239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:36.853241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:36.853363 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:36.853745 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:36.853754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:36.853755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:36.853947 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:36.854064 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:36.854668 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 02:11:36.854811 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:11:36.855983 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:36.840967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36327 (local address 127.28.21.1:52789)
0806 02:11:36.841471 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:36.841528 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:36.842939 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:36.843544 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:36.843555 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:36.844119 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:36.844839 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:36.844858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.845897 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.845901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:36.847554 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:36.847570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:36.848274 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:36.848278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:36.848439 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:36.849134 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:36.849152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:36.850897 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:36.852793 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:36.852808 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:36.852824 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:36.853106 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:36.853471 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:36.853476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:36.853478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:36.853631 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:36.854062 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:36.854070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:36.854332 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 02:11:36.854602 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 02:11:36.854964 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.2@KRBTEST.COM" created.
I20260806 02:11:36.857493 16883 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36327
I20260806 02:11:36.857762 16883 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:36.858366 16883 heartbeater.cc:511] Master 127.28.21.62:36327 requested a full tablet report, sending...
I20260806 02:11:36.860540 16679 ts_manager.cc:194] Registered new tserver with Master: fb74c5df21c34443b06b2a74ee426394 (127.28.21.1:45055)
I20260806 02:11:36.862231 16679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.1@KRBTEST.COM'} at 127.28.21.1:52789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
Entry for principal kudu/127.28.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab.
I20260806 02:11:36.873401 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:36.982901 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:36.983153 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:36.983196 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:36.987089 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:36.987142 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:36.987198 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:36.992305 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.2.keytab
--principal=kudu/127.28.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:36.993280 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:36.994218 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:37.003903 16897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:37.003938 16895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:37.004019 16889 server_base.cc:1034] running on GCE node
W20260806 02:11:37.003957 16894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:37.004549 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:37.005174 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:37.006361 16889 hybrid_clock.cc:648] HybridClock initialized: now 1785982297006344 us; error 33 us; skew 500 ppm
Aug 06 02:11:37 dist-test-slave-gp6n krb5kdc[16633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:37.009218 16889 init.cc:377] Logged in from keytab as kudu/127.28.21.2@KRBTEST.COM (short username kudu)
I20260806 02:11:37.010192 16889 webserver.cc:460] Webserver started at http://127.28.21.2:39799/ using document root <none> and password file <none>
I20260806 02:11:37.010813 16889 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:37.010902 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:37.011135 16889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:37.013130 16889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "5dc39941dbd84860b5c8381d6e5a921b"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1d1a3214d12b4dc8d2f829253ddebb2e"
server_key_iv: "6f60a2d882bd6910dfecb020b78bcf9e"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.013690 16889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "5dc39941dbd84860b5c8381d6e5a921b"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1d1a3214d12b4dc8d2f829253ddebb2e"
server_key_iv: "6f60a2d882bd6910dfecb020b78bcf9e"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.017652 16889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:11:37.020231 16904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.021477 16889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:37.021618 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "5dc39941dbd84860b5c8381d6e5a921b"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1d1a3214d12b4dc8d2f829253ddebb2e"
server_key_iv: "6f60a2d882bd6910dfecb020b78bcf9e"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.021734 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:37.036392 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:37.040239 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:37.040469 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:37.041096 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:37.041949 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:37.042021 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.042085 16889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:37.042116 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.051520 16889 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:42475
I20260806 02:11:37.051549 17025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:42475 every 8 connection(s)
I20260806 02:11:37.052418 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:37.060160 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 16889
I20260806 02:11:37.060304 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:37.060665 28756 external_mini_cluster.cc:1428] Setting key 3730183efb0167e2f8d2030f17f49104
Aug 06 02:11:37 dist-test-slave-gp6n krb5kdc[16633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.2@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:37.067458 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.054273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:52313 (local address 127.28.21.62:36327)
0806 02:11:37.054475 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.054480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.055418 (+ 938us) server_negotiation.cc:407] Connection header received
0806 02:11:37.056746 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.056749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.056806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.056887 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:37.058024 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.058633 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.059774 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.060075 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.060187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.063345 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:37.063348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:37.063350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:37.063373 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:37.065150 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.065711 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.065717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.065722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.065815 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.066144 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.066150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.066155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.066449 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:37.066555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.067088 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.067240 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:37.068216 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.054805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36327 (local address 127.28.21.2:52313)
0806 02:11:37.055270 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.055313 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.056534 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.057049 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.057056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.057404 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:37.057874 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.057887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.058755 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.058758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.059650 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.059659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.060228 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.060232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.060442 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.061170 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:37.061186 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:37.063141 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:37.065276 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:37.065320 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:37.065332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:37.065598 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:37.065911 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:37.065914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:37.065915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:37.066051 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:37.066567 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:37.066573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:37.066825 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.066993 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.067249 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260806 02:11:37.069454 17026 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36327
I20260806 02:11:37.069689 17026 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:37.070184 17026 heartbeater.cc:511] Master 127.28.21.62:36327 requested a full tablet report, sending...
I20260806 02:11:37.071338 16679 ts_manager.cc:194] Registered new tserver with Master: 5dc39941dbd84860b5c8381d6e5a921b (127.28.21.2:42475)
WARNING: no policy specified for kudu/127.28.21.3@KRBTEST.COM; defaulting to no policy
I20260806 02:11:37.071961 16679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.2@KRBTEST.COM'} at 127.28.21.2:52313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
Entry for principal kudu/127.28.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab.
I20260806 02:11:37.090449 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:37.201800 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:37.202085 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:37.202159 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:37.206334 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:37.206434 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:37.206521 17031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:37.212321 17031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.3.keytab
--principal=kudu/127.28.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:37.213500 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:37.214649 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:37.224500 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:37.224519 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:37.224491 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:37.224728 17031 server_base.cc:1034] running on GCE node
I20260806 02:11:37.225201 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:37.225833 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:37.227012 17031 hybrid_clock.cc:648] HybridClock initialized: now 1785982297226996 us; error 47 us; skew 500 ppm
Aug 06 02:11:37 dist-test-slave-gp6n krb5kdc[16633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:37.229945 17031 init.cc:377] Logged in from keytab as kudu/127.28.21.3@KRBTEST.COM (short username kudu)
I20260806 02:11:37.230876 17031 webserver.cc:460] Webserver started at http://127.28.21.3:39925/ using document root <none> and password file <none>
I20260806 02:11:37.231434 17031 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:37.231495 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:37.231701 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:37.233683 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "d62648c856784b3181d6b1da677d6931"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1bbfab4d08ec9d60fd44b7c0756f2e85"
server_key_iv: "717166d47b54aed1a4a5bb948618804c"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.234238 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "d62648c856784b3181d6b1da677d6931"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1bbfab4d08ec9d60fd44b7c0756f2e85"
server_key_iv: "717166d47b54aed1a4a5bb948618804c"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.238085 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:11:37.240787 17046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.242069 17031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:11:37.242300 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "d62648c856784b3181d6b1da677d6931"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1bbfab4d08ec9d60fd44b7c0756f2e85"
server_key_iv: "717166d47b54aed1a4a5bb948618804c"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.242477 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:37.266865 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:37.270591 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:37.270761 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:37.271374 17031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:37.272212 17031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:37.272264 17031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.272315 17031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:37.272338 17031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.281723 17031 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:38587
I20260806 02:11:37.281745 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:38587 every 8 connection(s)
I20260806 02:11:37.282701 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:37.287607 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17031
I20260806 02:11:37.287755 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:37.288081 28756 external_mini_cluster.cc:1428] Setting key 3195816722c6b74ad76e9dea5f4504af
Aug 06 02:11:37 dist-test-slave-gp6n krb5kdc[16633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982297, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.3@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:37.296821 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.284661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:56583 (local address 127.28.21.62:36327)
0806 02:11:37.284798 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.284802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.285469 (+ 667us) server_negotiation.cc:407] Connection header received
0806 02:11:37.286819 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.286823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.286882 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.286967 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:37.288540 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.289315 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.290457 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.290712 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.290825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.293242 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:37.293245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:37.293248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:37.293274 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:37.294806 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.295312 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.295316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.295318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.295401 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.295747 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.295751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.295753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.295938 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:37.296056 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.296530 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.296652 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:11:37.297647 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.284852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:36327 (local address 127.28.21.3:56583)
0806 02:11:37.285299 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.285341 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.286602 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.287131 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.287139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.287661 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:37.288372 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.288398 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.289429 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.289433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.290312 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.290321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.290859 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.290863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.291037 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.291633 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:11:37.291649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:37.293086 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:37.294931 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:37.294937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:37.294951 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:37.295207 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:37.295507 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:11:37.295510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:11:37.295512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:11:37.295650 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:11:37.296065 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:37.296074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:37.296306 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.296490 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.296794 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
I20260806 02:11:37.298813 17168 heartbeater.cc:347] Connected to a master server at 127.28.21.62:36327
I20260806 02:11:37.299073 17168 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:37.299575 17168 heartbeater.cc:511] Master 127.28.21.62:36327 requested a full tablet report, sending...
I20260806 02:11:37.300709 16679 ts_manager.cc:194] Registered new tserver with Master: d62648c856784b3181d6b1da677d6931 (127.28.21.3:38587)
I20260806 02:11:37.301493 16679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.21.3@KRBTEST.COM'} at 127.28.21.3:56583
I20260806 02:11:37.302220 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:37.312537 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.304769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.28.21.62:36327)
0806 02:11:37.304907 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.304910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.305141 (+ 231us) server_negotiation.cc:407] Connection header received
0806 02:11:37.305241 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.305243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.305278 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.305347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:37.306016 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.306584 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.307394 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.307605 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.307689 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.309559 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:37.309562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:37.309563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:37.309585 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:37.311081 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.311445 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.311447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.311449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.311523 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:37.311752 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:37.311754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:37.311755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:37.311967 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:37.312070 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.312319 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.312421 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:11:37.315637 16678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:37.318670 16678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:37.336411 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:42475 (local address 127.0.0.1:40000)
0806 02:11:37.328047 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.328105 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.328246 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.329057 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.329063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.329084 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:37.329247 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.329252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.330803 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.330807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.334639 (+ 3832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.334655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.335856 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.335860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.335979 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.336016 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.336129 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.336258 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":127,"spinlock_wait_cycles":6144,"thread_start_us":96,"threads_started":1}
I20260806 02:11:37.336419 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.28.21.2:42475)
0806 02:11:37.328622 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.328632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.328652 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:11:37.328716 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.328721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.328897 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.329033 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:37.329375 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.330644 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.334801 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.335741 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.335859 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.336020 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.336169 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.336258 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":115,"threads_started":1}
I20260806 02:11:37.338717 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:38587 (local address 127.0.0.1:33026)
0806 02:11:37.329277 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.329306 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.329394 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.329854 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.329857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.329874 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:37.329952 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.329958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.331064 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.331067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.335744 (+ 4677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.335758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.338240 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.338244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.338355 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.338380 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.338493 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.338581 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"mutex_wait_us":3918,"thread_start_us":105,"threads_started":1}
I20260806 02:11:37.338752 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.28.21.3:38587)
0806 02:11:37.328196 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.328210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.329502 (+ 1292us) server_negotiation.cc:407] Connection header received
0806 02:11:37.329576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.329580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.329725 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.329890 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:37.330043 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.330904 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.337103 (+ 6199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.338110 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.338239 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.338383 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.338522 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.338619 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":163,"threads_started":1}
I20260806 02:11:37.339306 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.28.21.1:45055)
0806 02:11:37.328406 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.328424 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.328450 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:11:37.328537 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.328543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.328764 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.328925 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:37.329509 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.330671 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.337067 (+ 6396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.337881 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.338004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.338135 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.339073 (+ 938us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.339168 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":93,"threads_started":1}
I20260806 02:11:37.339677 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.327553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:45055 (local address 127.0.0.1:58558)
0806 02:11:37.328047 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.328105 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.328246 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.329119 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.329122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.329140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:37.329318 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.329330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.330820 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.330823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.336929 (+ 6106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.336943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.338201 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.338205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.338292 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.338314 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.338382 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.338477 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":4723,"spinlock_wait_cycles":4224,"thread_start_us":122,"threads_started":1}
I20260806 02:11:37.339943 16955 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d (DEFAULT_TABLE table=test-table [id=4eaba358e2494fdcaeb620e1da24c040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:37.341013 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3ca8dd644d42e4b486ba6d9584e72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:37.341845 17096 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d (DEFAULT_TABLE table=test-table [id=4eaba358e2494fdcaeb620e1da24c040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:37.343060 17096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3ca8dd644d42e4b486ba6d9584e72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:37.342929 16811 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d (DEFAULT_TABLE table=test-table [id=4eaba358e2494fdcaeb620e1da24c040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:37.344652 16811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3ca8dd644d42e4b486ba6d9584e72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:37.347798 17185 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Bootstrap starting.
I20260806 02:11:37.350495 17185 tablet_bootstrap.cc:654] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:37.350623 17186 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Bootstrap starting.
I20260806 02:11:37.351418 17185 log.cc:824] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:37.352620 17186 tablet_bootstrap.cc:654] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:37.353008 17189 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Bootstrap starting.
I20260806 02:11:37.353746 17186 log.cc:824] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:37.355104 17189 tablet_bootstrap.cc:654] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:37.355538 17186 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: No bootstrap required, opened a new log
I20260806 02:11:37.355768 17186 ts_tablet_manager.cc:1397] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 02:11:37.355887 17185 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: No bootstrap required, opened a new log
I20260806 02:11:37.356115 17185 ts_tablet_manager.cc:1397] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:11:37.356437 17189 log.cc:824] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:37.358493 17189 tablet_bootstrap.cc:492] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: No bootstrap required, opened a new log
I20260806 02:11:37.358680 17189 ts_tablet_manager.cc:1397] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:11:37.359089 17186 raft_consensus.cc:348] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.359329 17186 raft_consensus.cc:374] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:37.359400 17186 raft_consensus.cc:729] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62648c856784b3181d6b1da677d6931, State: Initialized, Role: FOLLOWER
I20260806 02:11:37.359862 17186 consensus_queue.cc:260] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.360565 17168 heartbeater.cc:503] Master 127.28.21.62:36327 was elected leader, sending a full tablet report...
I20260806 02:11:37.360543 17185 raft_consensus.cc:348] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.360913 17185 raft_consensus.cc:374] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:37.360985 17185 raft_consensus.cc:729] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc39941dbd84860b5c8381d6e5a921b, State: Initialized, Role: FOLLOWER
I20260806 02:11:37.361479 17185 consensus_queue.cc:260] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.361965 17186 ts_tablet_manager.cc:1428] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:11:37.362561 17185 ts_tablet_manager.cc:1428] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:11:37.362820 17026 heartbeater.cc:503] Master 127.28.21.62:36327 was elected leader, sending a full tablet report...
I20260806 02:11:37.363775 17189 raft_consensus.cc:348] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.364054 17189 raft_consensus.cc:374] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:37.364113 17189 raft_consensus.cc:729] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb74c5df21c34443b06b2a74ee426394, State: Initialized, Role: FOLLOWER
I20260806 02:11:37.364568 17189 consensus_queue.cc:260] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.365427 16883 heartbeater.cc:503] Master 127.28.21.62:36327 was elected leader, sending a full tablet report...
I20260806 02:11:37.365748 17189 ts_tablet_manager.cc:1428] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260806 02:11:37.533771 17169 tablet.cc:2369] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:11:37.553571 17027 tablet.cc:2369] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:37.555238 17194 raft_consensus.cc:482] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:37.555487 17194 raft_consensus.cc:504] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.556694 17194 leader_election.cc:288] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc39941dbd84860b5c8381d6e5a921b (127.28.21.2:42475), d62648c856784b3181d6b1da677d6931 (127.28.21.3:38587)
I20260806 02:11:37.562445 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.556982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:38587 (local address 127.28.21.1:58163)
0806 02:11:37.557304 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.557317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.557469 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.557768 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.557771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.557802 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:37.558103 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.558112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.559225 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.559229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.560833 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.560841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.561910 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.561913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.562057 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.562078 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.562147 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.562238 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":208,"spinlock_wait_cycles":16256,"thread_start_us":97,"threads_started":1}
I20260806 02:11:37.562461 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.557098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:58163 (local address 127.28.21.3:38587)
0806 02:11:37.557230 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.557236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.557425 (+ 189us) server_negotiation.cc:407] Connection header received
0806 02:11:37.557588 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.557591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.557645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.557730 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:37.558227 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.559053 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.560969 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.561689 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.561902 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.561966 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.562216 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.562304 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:11:37.562506 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.557040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:36967 (local address 127.28.21.2:42475)
0806 02:11:37.557178 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.557184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.557457 (+ 273us) server_negotiation.cc:407] Connection header received
0806 02:11:37.557588 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.557591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.557647 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.557732 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:37.558227 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.559049 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.560969 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.561696 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.561853 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.561929 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.562215 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.562348 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:37.562445 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.556902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:42475 (local address 127.28.21.1:36967)
0806 02:11:37.557301 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.557317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:37.557469 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:37.557768 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:37.557771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:37.557802 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:37.558103 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.558112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.559225 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.559229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.560829 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:37.560841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.561853 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.561856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:37.561977 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.562047 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:11:37.562133 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:11:37.562238 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":314,"spinlock_wait_cycles":36864,"thread_start_us":139,"threads_started":1}
I20260806 02:11:37.563289 17117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3ca8dd644d42e4b486ba6d9584e72d" candidate_uuid: "fb74c5df21c34443b06b2a74ee426394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62648c856784b3181d6b1da677d6931" is_pre_election: true
I20260806 02:11:37.563356 16976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3ca8dd644d42e4b486ba6d9584e72d" candidate_uuid: "fb74c5df21c34443b06b2a74ee426394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc39941dbd84860b5c8381d6e5a921b" is_pre_election: true
I20260806 02:11:37.563617 17117 raft_consensus.cc:2454] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb74c5df21c34443b06b2a74ee426394 in term 0.
I20260806 02:11:37.563629 16976 raft_consensus.cc:2454] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb74c5df21c34443b06b2a74ee426394 in term 0.
I20260806 02:11:37.564123 16760 leader_election.cc:302] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc39941dbd84860b5c8381d6e5a921b, fb74c5df21c34443b06b2a74ee426394; no voters:
I20260806 02:11:37.564435 17194 raft_consensus.cc:2781] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:37.564540 17194 raft_consensus.cc:482] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:37.564596 17194 raft_consensus.cc:3037] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:37.565713 17194 raft_consensus.cc:504] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.566085 17194 leader_election.cc:288] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc39941dbd84860b5c8381d6e5a921b (127.28.21.2:42475), d62648c856784b3181d6b1da677d6931 (127.28.21.3:38587)
I20260806 02:11:37.566476 16976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3ca8dd644d42e4b486ba6d9584e72d" candidate_uuid: "fb74c5df21c34443b06b2a74ee426394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc39941dbd84860b5c8381d6e5a921b"
I20260806 02:11:37.566512 17117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3ca8dd644d42e4b486ba6d9584e72d" candidate_uuid: "fb74c5df21c34443b06b2a74ee426394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62648c856784b3181d6b1da677d6931"
I20260806 02:11:37.566610 16976 raft_consensus.cc:3037] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:37.566642 17117 raft_consensus.cc:3037] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:37.567646 16976 raft_consensus.cc:2454] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb74c5df21c34443b06b2a74ee426394 in term 1.
I20260806 02:11:37.567713 17117 raft_consensus.cc:2454] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb74c5df21c34443b06b2a74ee426394 in term 1.
I20260806 02:11:37.567958 16760 leader_election.cc:302] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc39941dbd84860b5c8381d6e5a921b, fb74c5df21c34443b06b2a74ee426394; no voters:
I20260806 02:11:37.568111 17194 raft_consensus.cc:2781] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:37.568343 17194 raft_consensus.cc:686] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 1 LEADER]: Becoming Leader. State: Replica: fb74c5df21c34443b06b2a74ee426394, State: Running, Role: LEADER
I20260806 02:11:37.568712 17194 consensus_queue.cc:237] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } }
I20260806 02:11:37.571825 16678 catalog_manager.cc:5495] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb74c5df21c34443b06b2a74ee426394 (127.28.21.1). New cstate: current_term: 1 leader_uuid: "fb74c5df21c34443b06b2a74ee426394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb74c5df21c34443b06b2a74ee426394" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 45055 } health_report { overall_health: HEALTHY } } }
W20260806 02:11:37.589936 16884 tablet.cc:2369] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:37.624212 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.620627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.28.21.1:45055)
0806 02:11:37.620759 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.620763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.620779 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:11:37.620883 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.620885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.620945 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.621065 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:37.621302 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.621809 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.622752 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.622975 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.623082 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.623381 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:37.623807 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:37.623940 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.624007 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.624064 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:11:37.630204 17117 raft_consensus.cc:1261] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 1 FOLLOWER]: Refusing update from remote peer fb74c5df21c34443b06b2a74ee426394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:37.630216 16976 raft_consensus.cc:1261] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 1 FOLLOWER]: Refusing update from remote peer fb74c5df21c34443b06b2a74ee426394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:37.631065 17194 consensus_queue.cc:1035] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62648c856784b3181d6b1da677d6931" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:37.631409 17197 consensus_queue.cc:1035] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc39941dbd84860b5c8381d6e5a921b" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:37.639514 17202 mvcc.cc:205] Tried to move back new op lower bound from 7315383491085160448 to 7315383490844057600. Current Snapshot: MvccSnapshot[applied={T|T < 7315383491085160448}]
I20260806 02:11:37.642148 17201 mvcc.cc:205] Tried to move back new op lower bound from 7315383491085160448 to 7315383490844057600. Current Snapshot: MvccSnapshot[applied={T|T < 7315383491085160448 or (T in {7315383491085160448})}]
I20260806 02:11:37.647300 16678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:37.647485 16678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:37.650003 16678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e42a937a9ca441ca08ddc5a97e3cefd: Sending DeleteTablet for 3 replicas of tablet 6d3ca8dd644d42e4b486ba6d9584e72d
I20260806 02:11:37.650759 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:37 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260806 02:11:37.650898 16811 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:37 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:58558
I20260806 02:11:37.651126 17096 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d3ca8dd644d42e4b486ba6d9584e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:37 UTC) from {username='kudu', principal='kudu/127.28.21.62@KRBTEST.COM'} at 127.0.0.1:33026
I20260806 02:11:37.651340 17209 tablet_replica.cc:330] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: stopping tablet replica
I20260806 02:11:37.651589 17211 tablet_replica.cc:330] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: stopping tablet replica
I20260806 02:11:37.651634 17209 raft_consensus.cc:2229] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:37.651913 17211 raft_consensus.cc:2229] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:37.651968 17209 raft_consensus.cc:2258] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:37.652194 17210 tablet_replica.cc:330] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394: stopping tablet replica
I20260806 02:11:37.652278 17211 raft_consensus.cc:2258] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:37.652309 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16744
I20260806 02:11:37.652555 17210 raft_consensus.cc:2229] T 6d3ca8dd644d42e4b486ba6d9584e72d P fb74c5df21c34443b06b2a74ee426394 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:11:37.654372 17211 ts_tablet_manager.cc:1905] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:37.656060 17209 ts_tablet_manager.cc:1905] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:37.658089 17209 ts_tablet_manager.cc:1918] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:37.658203 17209 log.cc:1196] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-1/wal/wals/6d3ca8dd644d42e4b486ba6d9584e72d
I20260806 02:11:37.658196 17211 ts_tablet_manager.cc:1918] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:37.658295 17211 log.cc:1196] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/6d3ca8dd644d42e4b486ba6d9584e72d
I20260806 02:11:37.658710 17211 ts_tablet_manager.cc:1939] T 6d3ca8dd644d42e4b486ba6d9584e72d P d62648c856784b3181d6b1da677d6931: Deleting consensus metadata
I20260806 02:11:37.658766 17209 ts_tablet_manager.cc:1939] T 6d3ca8dd644d42e4b486ba6d9584e72d P 5dc39941dbd84860b5c8381d6e5a921b: Deleting consensus metadata
I20260806 02:11:37.659981 16662 catalog_manager.cc:4841] TS 5dc39941dbd84860b5c8381d6e5a921b (127.28.21.2:42475): tablet 6d3ca8dd644d42e4b486ba6d9584e72d (table test-table [id=4eaba358e2494fdcaeb620e1da24c040]) successfully deleted
I20260806 02:11:37.660049 16663 catalog_manager.cc:4841] TS d62648c856784b3181d6b1da677d6931 (127.28.21.3:38587): tablet 6d3ca8dd644d42e4b486ba6d9584e72d (table test-table [id=4eaba358e2494fdcaeb620e1da24c040]) successfully deleted
W20260806 02:11:37.661157 16664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:45055 (error 108)
I20260806 02:11:37.661947 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16889
I20260806 02:11:37.662226 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.661877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:45055 (local address 127.0.0.1:58576)
0806 02:11:37.662053 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:37.662133 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 02:11:37.665036 16664 catalog_manager.cc:4584] TS fb74c5df21c34443b06b2a74ee426394 (127.28.21.1:45055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d3ca8dd644d42e4b486ba6d9584e72d: Network error: Client connection negotiation failed: client connection to 127.28.21.1:45055: connect: Connection refused (error 111)
I20260806 02:11:37.669437 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17031
I20260806 02:11:37.675725 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 16646
2026-08-06T02:11:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2559 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:37Z Disabled control of system clock
I20260806 02:11:37.707432 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42287
--webserver_interface=127.28.21.62
--webserver_port=0
--builtin_ntp_servers=127.28.21.20:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:37.815788 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:37.816025 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:37.816059 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:37.819923 17216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:37.819974 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:37.819989 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:37.820015 17216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:37.820029 17216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:37.824755 17216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:42287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:37.825811 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:37.827178 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:37.835628 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:37.835685 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:37.835798 17216 server_base.cc:1034] running on GCE node
W20260806 02:11:37.835945 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:37.836467 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:37.837477 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:37.838632 17216 hybrid_clock.cc:648] HybridClock initialized: now 1785982297838614 us; error 29 us; skew 500 ppm
I20260806 02:11:37.840449 17216 webserver.cc:460] Webserver started at http://127.28.21.62:39205/ using document root <none> and password file <none>
I20260806 02:11:37.840955 17216 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:37.841012 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:37.841167 17216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:37.843190 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "5172882ff4ee4d12a6914a6f881d7640"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1265668d3f46a7fad9ec7390c8cecd19"
server_key_iv: "0a1780338963a398b24520065b4054fd"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.843683 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "5172882ff4ee4d12a6914a6f881d7640"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1265668d3f46a7fad9ec7390c8cecd19"
server_key_iv: "0a1780338963a398b24520065b4054fd"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.847131 17216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:37.849881 17230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:37.851162 17216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:37.851300 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "5172882ff4ee4d12a6914a6f881d7640"
format_stamp: "Formatted at 2026-08-06 02:11:37 on dist-test-slave-gp6n"
server_key: "1265668d3f46a7fad9ec7390c8cecd19"
server_key_iv: "0a1780338963a398b24520065b4054fd"
server_key_version: "encryptionkey@0"
I20260806 02:11:37.851406 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:37.868716 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:37.869453 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:37.869669 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:37.877126 17216 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:42287
I20260806 02:11:37.877143 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:42287 every 8 connection(s)
I20260806 02:11:37.878086 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:37.881208 17284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:37.882932 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17216
I20260806 02:11:37.883059 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:37.883340 28756 external_mini_cluster.cc:1428] Setting key 384f4ca7156c8dd0f3c659bae2e4e733
I20260806 02:11:37.888105 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: Bootstrap starting.
I20260806 02:11:37.890727 17284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:37.891463 17284 log.cc:824] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:37.893544 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:37.884771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.28.21.62:42287)
0806 02:11:37.885427 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:37.885448 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:37.885491 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:11:37.886255 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:37.886279 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:37.886793 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:37.887175 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:37.887598 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.888997 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.889891 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:37.890604 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:37.890777 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:37.891608 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:37.891614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:37.891628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:37.892091 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:37.892169 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:37.892196 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:37.892293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:37.892416 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:11:37.892671 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260806 02:11:37.893615 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: No bootstrap required, opened a new log
I20260806 02:11:37.897454 17284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } }
I20260806 02:11:37.897688 17284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:37.897723 17284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5172882ff4ee4d12a6914a6f881d7640, State: Initialized, Role: FOLLOWER
I20260806 02:11:37.898072 17284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } }
I20260806 02:11:37.898177 17284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:37.898214 17284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:37.898288 17284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:37.899166 17284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } }
I20260806 02:11:37.899459 17284 leader_election.cc:302] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5172882ff4ee4d12a6914a6f881d7640; no voters:
I20260806 02:11:37.899703 17284 leader_election.cc:288] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:37.899881 17289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:37.900137 17289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [term 1 LEADER]: Becoming Leader. State: Replica: 5172882ff4ee4d12a6914a6f881d7640, State: Running, Role: LEADER
I20260806 02:11:37.900482 17289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } }
I20260806 02:11:37.900892 17284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:37.902122 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5172882ff4ee4d12a6914a6f881d7640. Latest consensus state: current_term: 1 leader_uuid: "5172882ff4ee4d12a6914a6f881d7640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } } }
I20260806 02:11:37.902257 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:37.902540 17290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5172882ff4ee4d12a6914a6f881d7640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5172882ff4ee4d12a6914a6f881d7640" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 42287 } } }
I20260806 02:11:37.902724 17290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:37.902833 17301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:37.905923 17301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:37.911653 17301 catalog_manager.cc:1347] Generated new cluster ID: 463711e61bf84c9cb3cd1dba4724ada8
I20260806 02:11:37.911716 17301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:37.951021 17301 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:37.952292 17301 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:37.964838 17301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: Generated new TSK 0
I20260806 02:11:37.965668 17301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:37.971920 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.1:0
--local_ip_for_outbound_sockets=127.28.21.1
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--builtin_ntp_servers=127.28.21.20:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:11:38.085644 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:38.085896 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:38.085932 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:38.089901 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:38.089954 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:38.090008 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.1
I20260806 02:11:38.094650 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.21.1
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.1
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:38.095659 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:38.096983 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:38.106881 17319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:38.106957 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:38.106992 17310 server_base.cc:1034] running on GCE node
W20260806 02:11:38.106918 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:38.107745 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:38.108394 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:38.109588 17310 hybrid_clock.cc:648] HybridClock initialized: now 1785982298109573 us; error 26 us; skew 500 ppm
I20260806 02:11:38.111472 17310 webserver.cc:460] Webserver started at http://127.28.21.1:36449/ using document root <none> and password file <none>
I20260806 02:11:38.112066 17310 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:38.112159 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:38.112389 17310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:38.114454 17310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data/instance:
uuid: "6996136de4074254b202d237f6e870a1"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "8e7f15b760071e6da3fc60a7dcd3d069"
server_key_iv: "020893cb17c7014ed3b1b96270694ada"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.115056 17310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance:
uuid: "6996136de4074254b202d237f6e870a1"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "8e7f15b760071e6da3fc60a7dcd3d069"
server_key_iv: "020893cb17c7014ed3b1b96270694ada"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.118805 17310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:11:38.121383 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.122746 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:38.122879 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal
uuid: "6996136de4074254b202d237f6e870a1"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "8e7f15b760071e6da3fc60a7dcd3d069"
server_key_iv: "020893cb17c7014ed3b1b96270694ada"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.122993 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:38.154731 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:38.155464 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:38.155674 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:38.156332 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:38.157316 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:38.157397 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.157474 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:38.157533 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.166519 17310 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.1:37373
I20260806 02:11:38.166657 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.1:37373 every 8 connection(s)
I20260806 02:11:38.167454 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/data/info.pb
I20260806 02:11:38.169389 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17310
I20260806 02:11:38.169525 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-0/wal/instance
I20260806 02:11:38.169835 28756 external_mini_cluster.cc:1428] Setting key a4553f9d4a2d344789d64a8df6f9fa43
I20260806 02:11:38.172016 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.2:0
--local_ip_for_outbound_sockets=127.28.21.2
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--builtin_ntp_servers=127.28.21.20:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:38.179080 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.169447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.1:46331 (local address 127.28.21.62:42287)
0806 02:11:38.169623 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.169629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.171206 (+ 1577us) server_negotiation.cc:407] Connection header received
0806 02:11:38.172898 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.172902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.172959 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.173071 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:38.174006 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.174664 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.176708 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.177001 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.177115 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.178574 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:38.178577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:38.178579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:38.178591 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:38.178609 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:38.178617 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:38.178678 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.178894 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.178949 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:11:38.180459 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.169949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42287 (local address 127.28.21.1:46331)
0806 02:11:38.171017 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.171076 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.172673 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.173252 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.173259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.173350 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:38.173858 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.173872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.174810 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.174817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.176575 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.176592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.177203 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.177211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.177482 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.178198 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:38.178203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:38.178270 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:38.178370 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:38.178712 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:38.178724 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:38.178735 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.178903 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.179254 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":132,"threads_started":1}
I20260806 02:11:38.181699 17442 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42287
I20260806 02:11:38.181989 17442 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:38.182742 17442 heartbeater.cc:511] Master 127.28.21.62:42287 requested a full tablet report, sending...
I20260806 02:11:38.184338 17247 ts_manager.cc:194] Registered new tserver with Master: 6996136de4074254b202d237f6e870a1 (127.28.21.1:37373)
I20260806 02:11:38.185492 17247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.1:46331
W20260806 02:11:38.291683 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:38.291975 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:38.292048 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:38.295957 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:38.296033 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:38.296120 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.2
I20260806 02:11:38.300693 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.21.2
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.2
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:38.301775 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:38.303272 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:38.312763 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:38.312753 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:38.312878 17446 server_base.cc:1034] running on GCE node
W20260806 02:11:38.312729 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:38.313282 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:38.313902 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:38.315060 17446 hybrid_clock.cc:648] HybridClock initialized: now 1785982298315032 us; error 39 us; skew 500 ppm
I20260806 02:11:38.317226 17446 webserver.cc:460] Webserver started at http://127.28.21.2:40393/ using document root <none> and password file <none>
I20260806 02:11:38.317790 17446 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:38.317854 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:38.318064 17446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:38.320132 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data/instance:
uuid: "e9e70bd6e077411d9c22a34d9302e908"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "4152c83517f37a16de5953c1c69d57ba"
server_key_iv: "20f9215c300763dee415844d8c486183"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.320709 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance:
uuid: "e9e70bd6e077411d9c22a34d9302e908"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "4152c83517f37a16de5953c1c69d57ba"
server_key_iv: "20f9215c300763dee415844d8c486183"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.324373 17446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:11:38.326947 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.328207 17446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:38.328380 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal
uuid: "e9e70bd6e077411d9c22a34d9302e908"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "4152c83517f37a16de5953c1c69d57ba"
server_key_iv: "20f9215c300763dee415844d8c486183"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.328500 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:38.346554 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:38.347265 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:38.347477 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:38.348062 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:38.348989 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:38.349061 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.349134 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:38.349186 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.358523 17446 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.2:37975
I20260806 02:11:38.358551 17578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.2:37975 every 8 connection(s)
I20260806 02:11:38.359411 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/data/info.pb
I20260806 02:11:38.359637 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17446
I20260806 02:11:38.359776 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-1/wal/instance
I20260806 02:11:38.360119 28756 external_mini_cluster.cc:1428] Setting key 6b78e21f3dd9503cf47379ebecb77d90
I20260806 02:11:38.362238 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.21.3:0
--local_ip_for_outbound_sockets=127.28.21.3
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--builtin_ntp_servers=127.28.21.20:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:11:38.369994 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.361295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:60795 (local address 127.28.21.62:42287)
0806 02:11:38.361428 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.361432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.362762 (+ 1330us) server_negotiation.cc:407] Connection header received
0806 02:11:38.364143 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.364147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.364200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.364281 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:38.365258 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.365831 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.367777 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.368027 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.368131 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.369460 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:38.369463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:38.369465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:38.369477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:38.369501 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:38.369509 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:38.369568 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.369810 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.369880 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:11:38.371265 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.361854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42287 (local address 127.28.21.2:60795)
0806 02:11:38.362624 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.362676 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.363943 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.364452 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.364459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.364541 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:38.365099 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.365121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.365980 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.365987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.367631 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.367646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.368211 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.368218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.368452 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.369126 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:38.369130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:38.369189 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:38.369291 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:38.369602 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:38.369615 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:38.369624 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.369774 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.370107 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260806 02:11:38.372540 17579 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42287
I20260806 02:11:38.372777 17579 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:38.373282 17579 heartbeater.cc:511] Master 127.28.21.62:42287 requested a full tablet report, sending...
I20260806 02:11:38.374311 17247 ts_manager.cc:194] Registered new tserver with Master: e9e70bd6e077411d9c22a34d9302e908 (127.28.21.2:37975)
I20260806 02:11:38.374897 17247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.2:60795
W20260806 02:11:38.476246 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:38.476526 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:38.476590 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:38.480676 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:38.480753 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:38.480839 17582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.21.3
I20260806 02:11:38.485575 17582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.21.3
--webserver_port=0
--tserver_master_addrs=127.28.21.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.21.3
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:38.486676 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:38.488014 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:38.497684 17587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:38.497730 17588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:38.497720 17590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:38.498222 17582 server_base.cc:1034] running on GCE node
I20260806 02:11:38.498700 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:38.499366 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:38.500558 17582 hybrid_clock.cc:648] HybridClock initialized: now 1785982298500542 us; error 26 us; skew 500 ppm
I20260806 02:11:38.502755 17582 webserver.cc:460] Webserver started at http://127.28.21.3:34835/ using document root <none> and password file <none>
I20260806 02:11:38.503387 17582 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:38.503486 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:38.503737 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:38.505923 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data/instance:
uuid: "d7d0e56561f546b0bb9c57a2e1d18314"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "b4c82da9dff0cffaaec97a87fe425540"
server_key_iv: "cfac6d4af4524889a34ca5c7af97385d"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.506583 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d0e56561f546b0bb9c57a2e1d18314"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "b4c82da9dff0cffaaec97a87fe425540"
server_key_iv: "cfac6d4af4524889a34ca5c7af97385d"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.510454 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:11:38.512920 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.514185 17582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:38.514364 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal
uuid: "d7d0e56561f546b0bb9c57a2e1d18314"
format_stamp: "Formatted at 2026-08-06 02:11:38 on dist-test-slave-gp6n"
server_key: "b4c82da9dff0cffaaec97a87fe425540"
server_key_iv: "cfac6d4af4524889a34ca5c7af97385d"
server_key_version: "encryptionkey@0"
I20260806 02:11:38.514542 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:38.537555 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:38.538249 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:38.538489 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:38.539095 17582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:11:38.540082 17582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:11:38.540160 17582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.540233 17582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:11:38.540292 17582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:38.548925 17582 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.3:35177
I20260806 02:11:38.548960 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.3:35177 every 8 connection(s)
I20260806 02:11:38.549854 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/data/info.pb
I20260806 02:11:38.559415 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.551714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.3:54621 (local address 127.28.21.62:42287)
0806 02:11:38.551848 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.551852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.552911 (+ 1059us) server_negotiation.cc:407] Connection header received
0806 02:11:38.554631 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.554634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.554692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.554801 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:38.555741 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.556288 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.557432 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.557677 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.557792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.558907 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:38.558911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:38.558913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:38.558934 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:38.558963 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:38.558971 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:38.559050 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.559210 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.559271 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:11:38.559532 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17582
I20260806 02:11:38.559656 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal/instance
I20260806 02:11:38.559984 28756 external_mini_cluster.cc:1428] Setting key 9ee20783f5dae5d084e350add4687f6a
I20260806 02:11:38.560359 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.552139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:42287 (local address 127.28.21.3:54621)
0806 02:11:38.552769 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.552808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.554230 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.554974 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.554980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.555063 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:11:38.555596 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.555612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.556419 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.556430 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.557304 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.557313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.557832 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.557836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.558001 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.558657 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:11:38.558659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:11:38.558697 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:11:38.558756 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:11:38.559052 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:11:38.559063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:11:38.559071 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.559197 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.559523 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260806 02:11:38.561518 17718 heartbeater.cc:347] Connected to a master server at 127.28.21.62:42287
I20260806 02:11:38.561755 17718 heartbeater.cc:464] Registering TS with master...
I20260806 02:11:38.562297 17718 heartbeater.cc:511] Master 127.28.21.62:42287 requested a full tablet report, sending...
I20260806 02:11:38.563369 17247 ts_manager.cc:194] Registered new tserver with Master: d7d0e56561f546b0bb9c57a2e1d18314 (127.28.21.3:35177)
I20260806 02:11:38.563994 17247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.21.3:54621
I20260806 02:11:38.573793 28756 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:11:38.579452 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.575801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.28.21.62:42287)
0806 02:11:38.575949 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.575954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.576108 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:11:38.576205 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.576207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.576247 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.576311 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:38.576548 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.577145 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.577952 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.578155 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.578227 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.579100 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:38.579103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:38.579105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:38.579114 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:38.579130 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:38.579137 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:38.579193 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.579243 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.579286 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:11:38.582190 17247 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:11:38.585597 17247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:11:38.599725 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.594455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:37373 (local address 127.0.0.1:42938)
0806 02:11:38.594995 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.595007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.595086 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.595871 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.595874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.595888 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:38.596017 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.596025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.597019 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.597025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.598311 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.598321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.599281 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.599284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.599373 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.599405 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.599515 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.599606 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":26,"thread_start_us":106,"threads_started":1}
I20260806 02:11:38.599852 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.594889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.28.21.1:37373)
0806 02:11:38.595405 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.595415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.595435 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:11:38.595498 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.595502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.595686 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.595886 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:38.596145 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.596899 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.598460 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.599142 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.599257 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.599459 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.599547 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.599658 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":120,"threads_started":1}
I20260806 02:11:38.600633 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.593693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37975 (local address 127.0.0.1:37110)
0806 02:11:38.594701 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.594749 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.594905 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.595657 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.595664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.595687 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:38.596041 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.596049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.597428 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.597432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.598809 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.598821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.600214 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.600218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.600304 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.600327 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.600405 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.600510 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":188,"thread_start_us":85,"threads_started":1}
I20260806 02:11:38.601426 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.593759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:35177 (local address 127.0.0.1:43718)
0806 02:11:38.594902 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.594915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.595014 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.595727 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.595730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.595746 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:38.595876 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.595886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.597428 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.597432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.598809 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.598821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.601005 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.601009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.601096 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.601118 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.601184 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.601247 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":91,"thread_start_us":103,"threads_started":1}
I20260806 02:11:38.601519 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.594166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.28.21.3:35177)
0806 02:11:38.595306 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.595318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.595352 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:11:38.595424 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.595428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.595582 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.595900 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:38.596047 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.597287 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.600139 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.600835 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.600984 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.601170 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.601219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.601337 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":137,"threads_started":1}
I20260806 02:11:38.601734 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.593814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.28.21.2:37975)
0806 02:11:38.594481 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.594500 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.595062 (+ 562us) server_negotiation.cc:407] Connection header received
0806 02:11:38.595268 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.595278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.595492 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.595658 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:38.596173 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.597294 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.598951 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.599754 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.599884 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.600081 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.600497 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.600595 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":133,"threads_started":1}
I20260806 02:11:38.603359 17375 tablet_service.cc:1467] Processing CreateTablet for tablet 287e39c97d3344a7976aafd82730a44c (DEFAULT_TABLE table=test-table [id=be27a3028c87402d881ed67bd4aced7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:38.604375 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287e39c97d3344a7976aafd82730a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:38.604543 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 287e39c97d3344a7976aafd82730a44c (DEFAULT_TABLE table=test-table [id=be27a3028c87402d881ed67bd4aced7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:38.605134 17646 tablet_service.cc:1467] Processing CreateTablet for tablet 287e39c97d3344a7976aafd82730a44c (DEFAULT_TABLE table=test-table [id=be27a3028c87402d881ed67bd4aced7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:11:38.605587 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287e39c97d3344a7976aafd82730a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:38.606122 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287e39c97d3344a7976aafd82730a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:38.610793 17735 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Bootstrap starting.
I20260806 02:11:38.612548 17736 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Bootstrap starting.
I20260806 02:11:38.613365 17735 tablet_bootstrap.cc:654] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:38.614441 17735 log.cc:824] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:38.614745 17737 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Bootstrap starting.
I20260806 02:11:38.615013 17736 tablet_bootstrap.cc:654] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:38.615993 17736 log.cc:824] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:38.616730 17737 tablet_bootstrap.cc:654] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:38.616955 17735 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: No bootstrap required, opened a new log
I20260806 02:11:38.617163 17735 ts_tablet_manager.cc:1397] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:11:38.617493 17737 log.cc:824] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:38.618384 17736 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: No bootstrap required, opened a new log
I20260806 02:11:38.618604 17736 ts_tablet_manager.cc:1397] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260806 02:11:38.619263 17737 tablet_bootstrap.cc:492] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: No bootstrap required, opened a new log
I20260806 02:11:38.619438 17737 ts_tablet_manager.cc:1397] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:11:38.622053 17735 raft_consensus.cc:348] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.622336 17735 raft_consensus.cc:374] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:38.622390 17735 raft_consensus.cc:729] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6996136de4074254b202d237f6e870a1, State: Initialized, Role: FOLLOWER
I20260806 02:11:38.622849 17737 raft_consensus.cc:348] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.622967 17735 consensus_queue.cc:260] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.623140 17737 raft_consensus.cc:374] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:38.623196 17737 raft_consensus.cc:729] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d0e56561f546b0bb9c57a2e1d18314, State: Initialized, Role: FOLLOWER
I20260806 02:11:38.623237 17736 raft_consensus.cc:348] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.623513 17736 raft_consensus.cc:374] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:38.623574 17736 raft_consensus.cc:729] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e70bd6e077411d9c22a34d9302e908, State: Initialized, Role: FOLLOWER
I20260806 02:11:38.623651 17737 consensus_queue.cc:260] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.624053 17735 ts_tablet_manager.cc:1428] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 02:11:38.623991 17736 consensus_queue.cc:260] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.624361 17718 heartbeater.cc:503] Master 127.28.21.62:42287 was elected leader, sending a full tablet report...
I20260806 02:11:38.624409 17442 heartbeater.cc:503] Master 127.28.21.62:42287 was elected leader, sending a full tablet report...
I20260806 02:11:38.624598 17737 ts_tablet_manager.cc:1428] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:11:38.624984 17736 ts_tablet_manager.cc:1428] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260806 02:11:38.625321 17579 heartbeater.cc:503] Master 127.28.21.62:42287 was elected leader, sending a full tablet report...
W20260806 02:11:38.668661 17443 tablet.cc:2369] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:11:38.801047 17719 tablet.cc:2369] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:11:38.860774 17580 tablet.cc:2369] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:11:38.868136 17744 raft_consensus.cc:482] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:11:38.868319 17744 raft_consensus.cc:504] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.869416 17744 leader_election.cc:288] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d0e56561f546b0bb9c57a2e1d18314 (127.28.21.3:35177), 6996136de4074254b202d237f6e870a1 (127.28.21.1:37373)
I20260806 02:11:38.874954 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.869887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:44739 (local address 127.28.21.3:35177)
0806 02:11:38.870018 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.870025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.870311 (+ 286us) server_negotiation.cc:407] Connection header received
0806 02:11:38.870508 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.870512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.870560 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.870640 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:38.870942 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.871668 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.873594 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.874242 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.874343 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.874434 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.874744 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.874818 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:11:38.874954 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.869881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.21.2:35427 (local address 127.28.21.1:37373)
0806 02:11:38.870020 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.870024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.870048 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:11:38.870257 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.870260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.870308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.870513 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:11:38.870822 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.871648 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.873594 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.874230 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.874381 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.874465 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.874744 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.874818 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:11:38.874987 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.869741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:37373 (local address 127.28.21.2:35427)
0806 02:11:38.869983 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.869998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.870102 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.870528 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.870530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.870544 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:38.870659 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.870666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.871822 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.871825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.873444 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.873456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.874383 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.874387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.874581 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.874625 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.874714 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.874813 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":169}
I20260806 02:11:38.874996 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.869741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.3:35177 (local address 127.28.21.2:44739)
0806 02:11:38.870205 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:38.870219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:11:38.870351 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:11:38.870665 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:11:38.870667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:11:38.870681 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:11:38.870831 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.870836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.871813 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.871817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.873444 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:11:38.873456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.874383 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.874387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:11:38.874577 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.874627 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:11:38.874713 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:11:38.874809 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":295,"thread_start_us":102,"threads_started":1}
I20260806 02:11:38.875686 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287e39c97d3344a7976aafd82730a44c" candidate_uuid: "e9e70bd6e077411d9c22a34d9302e908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" is_pre_election: true
I20260806 02:11:38.875697 17395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287e39c97d3344a7976aafd82730a44c" candidate_uuid: "e9e70bd6e077411d9c22a34d9302e908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996136de4074254b202d237f6e870a1" is_pre_election: true
I20260806 02:11:38.876009 17395 raft_consensus.cc:2454] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e70bd6e077411d9c22a34d9302e908 in term 0.
I20260806 02:11:38.875999 17668 raft_consensus.cc:2454] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e70bd6e077411d9c22a34d9302e908 in term 0.
I20260806 02:11:38.876485 17462 leader_election.cc:302] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d0e56561f546b0bb9c57a2e1d18314, e9e70bd6e077411d9c22a34d9302e908; no voters:
I20260806 02:11:38.876767 17744 raft_consensus.cc:2781] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:11:38.876946 17744 raft_consensus.cc:482] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:11:38.877017 17744 raft_consensus.cc:3037] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:38.878240 17744 raft_consensus.cc:504] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.878670 17744 leader_election.cc:288] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [CANDIDATE]: Term 1 election: Requested vote from peers d7d0e56561f546b0bb9c57a2e1d18314 (127.28.21.3:35177), 6996136de4074254b202d237f6e870a1 (127.28.21.1:37373)
I20260806 02:11:38.878993 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287e39c97d3344a7976aafd82730a44c" candidate_uuid: "e9e70bd6e077411d9c22a34d9302e908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d0e56561f546b0bb9c57a2e1d18314"
I20260806 02:11:38.879024 17395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287e39c97d3344a7976aafd82730a44c" candidate_uuid: "e9e70bd6e077411d9c22a34d9302e908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996136de4074254b202d237f6e870a1"
I20260806 02:11:38.879120 17668 raft_consensus.cc:3037] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:38.879144 17395 raft_consensus.cc:3037] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:38.880151 17395 raft_consensus.cc:2454] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e70bd6e077411d9c22a34d9302e908 in term 1.
I20260806 02:11:38.880370 17668 raft_consensus.cc:2454] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e70bd6e077411d9c22a34d9302e908 in term 1.
I20260806 02:11:38.880499 17461 leader_election.cc:302] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6996136de4074254b202d237f6e870a1, e9e70bd6e077411d9c22a34d9302e908; no voters:
I20260806 02:11:38.880689 17744 raft_consensus.cc:2781] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:38.880955 17744 raft_consensus.cc:686] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 1 LEADER]: Becoming Leader. State: Replica: e9e70bd6e077411d9c22a34d9302e908, State: Running, Role: LEADER
I20260806 02:11:38.881294 17744 consensus_queue.cc:237] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } }
I20260806 02:11:38.884490 17244 catalog_manager.cc:5495] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9e70bd6e077411d9c22a34d9302e908 (127.28.21.2). New cstate: current_term: 1 leader_uuid: "e9e70bd6e077411d9c22a34d9302e908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e70bd6e077411d9c22a34d9302e908" member_type: VOTER last_known_addr { host: "127.28.21.2" port: 37975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 } health_report { overall_health: UNKNOWN } } }
I20260806 02:11:38.961892 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:38.958451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.28.21.2:37975)
0806 02:11:38.958591 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:38.958595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:38.958610 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:11:38.958712 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:38.958715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:38.958773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:38.958863 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:11:38.959113 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.959644 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.960490 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:38.960695 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:38.960772 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:38.961029 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:11:38.961453 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:11:38.961543 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:38.961632 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:11:38.961679 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:11:38.968212 17668 raft_consensus.cc:1261] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 1 FOLLOWER]: Refusing update from remote peer e9e70bd6e077411d9c22a34d9302e908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:38.968216 17395 raft_consensus.cc:1261] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 1 FOLLOWER]: Refusing update from remote peer e9e70bd6e077411d9c22a34d9302e908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:11:38.968955 17744 consensus_queue.cc:1035] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d0e56561f546b0bb9c57a2e1d18314" member_type: VOTER last_known_addr { host: "127.28.21.3" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:38.969156 17747 consensus_queue.cc:1035] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6996136de4074254b202d237f6e870a1" member_type: VOTER last_known_addr { host: "127.28.21.1" port: 37373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:11:38.979543 17752 mvcc.cc:205] Tried to move back new op lower bound from 7315383496566374400 to 7315383496220332032. Current Snapshot: MvccSnapshot[applied={T|T < 7315383496566374400 or (T in {7315383496566374400})}]
I20260806 02:11:38.986855 17243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:38.987047 17243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:11:38.989626 17243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5172882ff4ee4d12a6914a6f881d7640: Sending DeleteTablet for 3 replicas of tablet 287e39c97d3344a7976aafd82730a44c
I20260806 02:11:38.990381 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 287e39c97d3344a7976aafd82730a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:38 UTC) from {username='slave'} at 127.0.0.1:37110
I20260806 02:11:38.990578 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet 287e39c97d3344a7976aafd82730a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:38 UTC) from {username='slave'} at 127.0.0.1:42938
I20260806 02:11:38.990613 17646 tablet_service.cc:1514] Processing DeleteTablet for tablet 287e39c97d3344a7976aafd82730a44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:11:38 UTC) from {username='slave'} at 127.0.0.1:43718
I20260806 02:11:38.991103 17762 tablet_replica.cc:330] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1: stopping tablet replica
I20260806 02:11:38.991441 17762 raft_consensus.cc:2229] T 287e39c97d3344a7976aafd82730a44c P 6996136de4074254b202d237f6e870a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:38.991562 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17310
I20260806 02:11:38.998083 17763 tablet_replica.cc:330] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: stopping tablet replica
I20260806 02:11:38.998503 17763 raft_consensus.cc:2229] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:11:38.998696 17760 tablet_replica.cc:330] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: stopping tablet replica
I20260806 02:11:38.998869 17763 raft_consensus.cc:2258] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:38.999095 17760 raft_consensus.cc:2229] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:11:38.999681 17760 raft_consensus.cc:2258] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:11:39.000754 17763 ts_tablet_manager.cc:1905] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:11:39.001026 17231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.21.1:37373 (error 108)
I20260806 02:11:39.001093 17760 ts_tablet_manager.cc:1905] T 287e39c97d3344a7976aafd82730a44c P e9e70bd6e077411d9c22a34d9302e908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:11:39.001981 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17446
I20260806 02:11:39.004703 17763 ts_tablet_manager.cc:1918] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:11:39.004843 17763 log.cc:1196] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Deleting WAL directory at /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785982043785074-28756-0/minicluster-data/ts-2/wal/wals/287e39c97d3344a7976aafd82730a44c
I20260806 02:11:39.005182 17763 ts_tablet_manager.cc:1939] T 287e39c97d3344a7976aafd82730a44c P d7d0e56561f546b0bb9c57a2e1d18314: Deleting consensus metadata
I20260806 02:11:39.005192 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:39.004804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.1:37373 (local address 127.0.0.1:42952)
0806 02:11:39.004992 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:39.005081 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.1:37373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 02:11:39.005389 17231 catalog_manager.cc:4584] TS 6996136de4074254b202d237f6e870a1 (127.28.21.1:37373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 287e39c97d3344a7976aafd82730a44c: Network error: Client connection negotiation failed: client connection to 127.28.21.1:37373: connect: Connection refused (error 111)
I20260806 02:11:39.006476 17232 catalog_manager.cc:4841] TS d7d0e56561f546b0bb9c57a2e1d18314 (127.28.21.3:35177): tablet 287e39c97d3344a7976aafd82730a44c (table test-table [id=be27a3028c87402d881ed67bd4aced7c]) successfully deleted
I20260806 02:11:39.009796 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17582
I20260806 02:11:39.011291 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:39.009951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.2:37975 (local address 127.0.0.1:37126)
0806 02:11:39.011119 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0806 02:11:39.011185 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.2:37975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1061}
I20260806 02:11:39.017534 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17216
2026-08-06T02:11:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1341 ms)
[----------] 6 tests from EncryptionPolicyTest (11030 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:39 dist-test-slave-gp6n krb5kdc[17767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:39 dist-test-slave-gp6n krb5kdc[17767](info): set up 2 sockets
Aug 06 02:11:39 dist-test-slave-gp6n krb5kdc[17767](info): commencing operation
krb5kdc: starting...
W20260806 02:11:40.093407 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:40.223677 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:40.337555 17780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:40.337831 17780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:40.337878 17780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:40.342167 17780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:40.342236 17780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:40.342257 17780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:40.342273 17780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:40.342288 17780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:40.347640 17780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:40.348732 17780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:40.349809 17780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:40.358693 17785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:40.358680 17788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:40.358675 17786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:40.359144 17780 server_base.cc:1034] running on GCE node
I20260806 02:11:40.359680 17780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:40.360877 17780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:40.362043 17780 hybrid_clock.cc:648] HybridClock initialized: now 1785982300362011 us; error 44 us; skew 500 ppm
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:40.364986 17780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:40.365895 17780 webserver.cc:460] Webserver started at http://127.0.0.1:40923/ using document root <none> and password file <none>
I20260806 02:11:40.366551 17780 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:40.366650 17780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:40.366891 17780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:40.368968 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "3532adf7e25544579335a8252d1aa31f"
format_stamp: "Formatted at 2026-08-06 02:11:40 on dist-test-slave-gp6n"
server_key: "46da18bd3a9c9178e5a4394e62b96b48"
server_key_iv: "8f173ee9b5a54ecc291fc0af78dc3ea1"
server_key_version: "encryptionkey@0"
I20260806 02:11:40.369578 17780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "3532adf7e25544579335a8252d1aa31f"
format_stamp: "Formatted at 2026-08-06 02:11:40 on dist-test-slave-gp6n"
server_key: "46da18bd3a9c9178e5a4394e62b96b48"
server_key_iv: "8f173ee9b5a54ecc291fc0af78dc3ea1"
server_key_version: "encryptionkey@0"
I20260806 02:11:40.373469 17780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:11:40.375988 17795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:40.377172 17780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:40.377315 17780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "3532adf7e25544579335a8252d1aa31f"
format_stamp: "Formatted at 2026-08-06 02:11:40 on dist-test-slave-gp6n"
server_key: "46da18bd3a9c9178e5a4394e62b96b48"
server_key_iv: "8f173ee9b5a54ecc291fc0af78dc3ea1"
server_key_version: "encryptionkey@0"
I20260806 02:11:40.377427 17780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:40.406247 17780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:40.409685 17780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:40.409919 17780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:40.417373 17780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40947
I20260806 02:11:40.417440 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40947 every 8 connection(s)
I20260806 02:11:40.418362 17780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:40.421247 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17780
I20260806 02:11:40.421335 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:40.421355 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:40.421660 28756 external_mini_cluster.cc:1428] Setting key 6cf0329710b6bb52cf8e136448934162
I20260806 02:11:40.429226 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: Bootstrap starting.
I20260806 02:11:40.431859 17853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: Neither blocks nor log segments found. Creating new log.
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:11:40.432850 17853 log.cc:824] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:40.434976 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: No bootstrap required, opened a new log
I20260806 02:11:40.438169 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:40.423242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56823 (local address 127.0.0.1:40947)
0806 02:11:40.423721 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:40.423745 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:40.423791 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:11:40.425104 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:40.425131 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:40.425607 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:40.426008 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:40.426832 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:40.428536 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:40.429488 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:40.429974 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:40.430158 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:40.433156 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:40.433162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:40.433174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:40.433201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:40.435005 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:40.435725 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:40.435729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:40.435733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:40.435846 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:40.436216 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:40.436219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:40.436220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:40.436624 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:40.436779 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:40.437154 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 02:11:40.437383 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
I20260806 02:11:40.439553 17853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:11:40.439849 17853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:40.439915 17853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3532adf7e25544579335a8252d1aa31f, State: Initialized, Role: FOLLOWER
I20260806 02:11:40.440433 17853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:11:40.440606 17853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:40.440744 17853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:40.440889 17853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:40.441378 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:40.442179 17853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:11:40.442601 17853 leader_election.cc:302] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3532adf7e25544579335a8252d1aa31f; no voters:
I20260806 02:11:40.442977 17853 leader_election.cc:288] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:40.443341 17863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:40.443622 17863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [term 1 LEADER]: Becoming Leader. State: Replica: 3532adf7e25544579335a8252d1aa31f, State: Running, Role: LEADER
I20260806 02:11:40.444000 17863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:11:40.444368 17853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:40.445683 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3532adf7e25544579335a8252d1aa31f. Latest consensus state: current_term: 1 leader_uuid: "3532adf7e25544579335a8252d1aa31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260806 02:11:40.446200 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:40.446882 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3532adf7e25544579335a8252d1aa31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532adf7e25544579335a8252d1aa31f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260806 02:11:40.446990 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f [sys.catalog]: This master's current role is: LEADER
W20260806 02:11:40.449885 17879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:40.450014 17879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:40.450120 17878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:40.451056 17878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:40.453569 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:40.444223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42559 (local address 127.0.0.1:40947)
0806 02:11:40.444314 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:40.444318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:40.444331 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:11:40.444371 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:40.444373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:40.444407 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:40.444502 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:40.445294 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:40.446214 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:40.447400 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:40.447759 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:40.447870 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:40.449856 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:40.449860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:40.449863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:40.449892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:40.451670 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:40.452192 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:40.452198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:40.452206 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:40.452282 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:40.452641 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:40.452644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:40.452646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:40.452832 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:40.452997 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:40.453291 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 02:11:40.453406 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 02:11:40.457376 17878 catalog_manager.cc:1347] Generated new cluster ID: cf48e4d955bc457a875ccfe71f84f6f3
I20260806 02:11:40.457437 17878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:40.473631 17878 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:40.474644 17878 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:40.486013 17878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3532adf7e25544579335a8252d1aa31f: Generated new TSK 0
I20260806 02:11:40.486845 17878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:40.490957 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17780
W20260806 02:11:40.498616 17765 connection.cc:535] client connection to 127.0.0.1:40947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40947): Connection reset by peer (error 104)
2026-08-06T02:11:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17885](info): set up 2 sockets
Aug 06 02:11:40 dist-test-slave-gp6n krb5kdc[17885](info): commencing operation
krb5kdc: starting...
W20260806 02:11:41.561157 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:41 dist-test-slave-gp6n krb5kdc[17885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:41.689472 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:41.799892 17898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:41.800220 17898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:41.800294 17898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:41.804328 17898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:41.804422 17898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:41.804462 17898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:41.804522 17898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:41.804553 17898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:41.809768 17898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:41.810897 17898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:41.811920 17898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:41.820484 17904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:41.820518 17903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:41.820466 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:41.820730 17898 server_base.cc:1034] running on GCE node
I20260806 02:11:41.821313 17898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:41.822481 17898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:41.823657 17898 hybrid_clock.cc:648] HybridClock initialized: now 1785982301823643 us; error 28 us; skew 500 ppm
Aug 06 02:11:41 dist-test-slave-gp6n krb5kdc[17885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:41.829949 17898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:41.830926 17898 webserver.cc:460] Webserver started at http://127.0.0.1:44529/ using document root <none> and password file <none>
I20260806 02:11:41.831461 17898 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:41.831528 17898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:41.831699 17898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:41.833707 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "ee76956020ee42f4bed6275d8ce132ea"
format_stamp: "Formatted at 2026-08-06 02:11:41 on dist-test-slave-gp6n"
server_key: "64ec246dc0bbac855c26dcf10b29ae48"
server_key_iv: "f07a338d225e346f581e1fdcf0b58b77"
server_key_version: "encryptionkey@0"
I20260806 02:11:41.834223 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "ee76956020ee42f4bed6275d8ce132ea"
format_stamp: "Formatted at 2026-08-06 02:11:41 on dist-test-slave-gp6n"
server_key: "64ec246dc0bbac855c26dcf10b29ae48"
server_key_iv: "f07a338d225e346f581e1fdcf0b58b77"
server_key_version: "encryptionkey@0"
I20260806 02:11:41.837843 17898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:11:41.840212 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:41.841449 17898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:11:41.841611 17898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "ee76956020ee42f4bed6275d8ce132ea"
format_stamp: "Formatted at 2026-08-06 02:11:41 on dist-test-slave-gp6n"
server_key: "64ec246dc0bbac855c26dcf10b29ae48"
server_key_iv: "f07a338d225e346f581e1fdcf0b58b77"
server_key_version: "encryptionkey@0"
I20260806 02:11:41.841723 17898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:41.854367 17898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:41.857242 17898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:41.857456 17898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:41.864900 17898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36547
I20260806 02:11:41.864925 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36547 every 8 connection(s)
I20260806 02:11:41.865873 17898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:41.868762 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:41.874655 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: Bootstrap starting.
I20260806 02:11:41.875980 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 17898
I20260806 02:11:41.876144 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:41.876464 28756 external_mini_cluster.cc:1428] Setting key 4ec60e47ea9186af760cf6db21038462
I20260806 02:11:41.877745 17969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:41.878654 17969 log.cc:824] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:41.880597 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: No bootstrap required, opened a new log
I20260806 02:11:41.884224 17969 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260806 02:11:41.884485 17969 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:41.884575 17969 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee76956020ee42f4bed6275d8ce132ea, State: Initialized, Role: FOLLOWER
I20260806 02:11:41.885070 17969 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260806 02:11:41.885263 17969 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:41.885329 17969 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:41.885429 17969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:41.885823 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:41.877663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35611 (local address 127.0.0.1:36547)
0806 02:11:41.878141 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:41.878162 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:41.878196 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:11:41.878951 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:41.878970 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:41.879417 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:41.879796 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:41.880220 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:41.881260 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:41.882165 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:41.882728 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:41.882902 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:41.883815 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:41.883820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:41.883832 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:41.884209 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:41.884262 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:41.884284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:41.884430 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:41.884700 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:11:41.884827 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260806 02:11:41.886549 17969 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260806 02:11:41.886895 17969 leader_election.cc:302] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee76956020ee42f4bed6275d8ce132ea; no voters:
I20260806 02:11:41.887233 17969 leader_election.cc:288] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:41.887331 17976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:41.887567 17976 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [term 1 LEADER]: Becoming Leader. State: Replica: ee76956020ee42f4bed6275d8ce132ea, State: Running, Role: LEADER
I20260806 02:11:41.887951 17976 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260806 02:11:41.888481 17969 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:41.888911 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:41.891475 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee76956020ee42f4bed6275d8ce132ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } } }
I20260806 02:11:41.892117 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:41.891824 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee76956020ee42f4bed6275d8ce132ea. Latest consensus state: current_term: 1 leader_uuid: "ee76956020ee42f4bed6275d8ce132ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee76956020ee42f4bed6275d8ce132ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } } }
I20260806 02:11:41.892403 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea [sys.catalog]: This master's current role is: LEADER
W20260806 02:11:41.894068 17996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:41.894198 17996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:41.894321 17999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:41.895216 17999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:41.895562 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:41.891717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59491 (local address 127.0.0.1:36547)
0806 02:11:41.891842 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:41.891847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:41.891875 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:11:41.891912 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:41.891918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:41.891967 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:41.892055 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:41.892331 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:41.893039 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:41.893879 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:41.894117 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:41.894225 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:41.895110 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:41.895114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:41.895117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:41.895129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:41.895161 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:41.895172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:41.895267 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:41.895392 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 02:11:41.895423 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:11:41.901250 17999 catalog_manager.cc:1347] Generated new cluster ID: 4cf1a60029ae4f20a734064bc3eda63a
I20260806 02:11:41.901327 17999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:41.924208 17999 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:41.925236 17999 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:41.931280 17999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee76956020ee42f4bed6275d8ce132ea: Generated new TSK 0
I20260806 02:11:41.931963 17999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:41.941989 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 17898
W20260806 02:11:41.949332 17883 connection.cc:535] client connection to 127.0.0.1:36547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36547): Connection reset by peer (error 104)
2026-08-06T02:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:41 dist-test-slave-gp6n krb5kdc[18004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:41 dist-test-slave-gp6n krb5kdc[18004](info): set up 2 sockets
Aug 06 02:11:41 dist-test-slave-gp6n krb5kdc[18004](info): commencing operation
krb5kdc: starting...
W20260806 02:11:43.024236 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:43 dist-test-slave-gp6n krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:43.153795 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:43.263078 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:43.263397 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:43.263484 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:43.267452 18017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:43.267552 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:43.267590 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:43.267643 18017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:43.267683 18017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:43.272992 18017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:43.274044 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:43.275107 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:43.283599 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:43.283592 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:43.283631 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:43.284060 18017 server_base.cc:1034] running on GCE node
I20260806 02:11:43.284693 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:43.285856 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:43.287060 18017 hybrid_clock.cc:648] HybridClock initialized: now 1785982303287043 us; error 35 us; skew 500 ppm
Aug 06 02:11:43 dist-test-slave-gp6n krb5kdc[18004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:43.289937 18017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:43.290866 18017 webserver.cc:460] Webserver started at http://127.0.0.1:35235/ using document root <none> and password file <none>
I20260806 02:11:43.291420 18017 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:43.291513 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:43.291745 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:43.293810 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "73ba1c258af04efa9f20a874bd71a9aa"
format_stamp: "Formatted at 2026-08-06 02:11:43 on dist-test-slave-gp6n"
server_key: "20ca3dece2ffcb23fda10932dc78d719"
server_key_iv: "89ae7c942608e620964d403907f7d146"
server_key_version: "encryptionkey@0"
I20260806 02:11:43.294464 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "73ba1c258af04efa9f20a874bd71a9aa"
format_stamp: "Formatted at 2026-08-06 02:11:43 on dist-test-slave-gp6n"
server_key: "20ca3dece2ffcb23fda10932dc78d719"
server_key_iv: "89ae7c942608e620964d403907f7d146"
server_key_version: "encryptionkey@0"
I20260806 02:11:43.298156 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:11:43.300587 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:43.301779 18017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:43.301920 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "73ba1c258af04efa9f20a874bd71a9aa"
format_stamp: "Formatted at 2026-08-06 02:11:43 on dist-test-slave-gp6n"
server_key: "20ca3dece2ffcb23fda10932dc78d719"
server_key_iv: "89ae7c942608e620964d403907f7d146"
server_key_version: "encryptionkey@0"
I20260806 02:11:43.302028 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:43.306553 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:43.309751 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:43.309943 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:43.317001 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38299
I20260806 02:11:43.317029 18085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38299 every 8 connection(s)
I20260806 02:11:43.317905 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:43.319872 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18017
I20260806 02:11:43.320022 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:43.320341 28756 external_mini_cluster.cc:1428] Setting key 0ae017c6c8d5e109d78b2318f652fd33
I20260806 02:11:43.320884 18086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:43.326098 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:43.321819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36759 (local address 127.0.0.1:38299)
0806 02:11:43.322260 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:43.322281 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:43.322325 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:11:43.323162 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:43.323185 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:43.323546 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:43.323927 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:43.324665 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:43.324670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:43.324685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:43.324697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:43.324764 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:43.324790 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:43.324946 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:43.325016 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:11:43.325176 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260806 02:11:43.328258 18086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa: Bootstrap starting.
I20260806 02:11:43.328876 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:43.330590 18086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:43.331480 18086 log.cc:824] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:43.332705 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:43.330961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48259 (local address 127.0.0.1:38299)
0806 02:11:43.331092 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:43.331097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:43.331336 (+ 239us) server_negotiation.cc:407] Connection header received
0806 02:11:43.331456 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:43.331458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:43.331485 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:43.331547 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:43.332256 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:43.332261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:43.332265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:43.332281 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:43.332311 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:43.332325 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:43.332431 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:43.332547 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:11:43.332576 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:11:43.333707 18086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa: No bootstrap required, opened a new log
I20260806 02:11:43.337131 18086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260806 02:11:43.337378 18086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:43.337472 18086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ba1c258af04efa9f20a874bd71a9aa, State: Initialized, Role: FOLLOWER
I20260806 02:11:43.337927 18086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260806 02:11:43.338079 18086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:43.338147 18086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:43.338243 18086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:43.339169 18086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260806 02:11:43.339510 18086 leader_election.cc:302] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ba1c258af04efa9f20a874bd71a9aa; no voters:
I20260806 02:11:43.339816 18086 leader_election.cc:288] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:43.339912 18097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:43.340185 18097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [term 1 LEADER]: Becoming Leader. State: Replica: 73ba1c258af04efa9f20a874bd71a9aa, State: Running, Role: LEADER
I20260806 02:11:43.340592 18097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } }
I20260806 02:11:43.340996 18086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:43.342111 18100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73ba1c258af04efa9f20a874bd71a9aa. Latest consensus state: current_term: 1 leader_uuid: "73ba1c258af04efa9f20a874bd71a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } } }
I20260806 02:11:43.342240 18100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:43.342370 18099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73ba1c258af04efa9f20a874bd71a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ba1c258af04efa9f20a874bd71a9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38299 } } }
I20260806 02:11:43.342483 18099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:43.343566 18107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:43.346133 18107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:43.351639 18107 catalog_manager.cc:1347] Generated new cluster ID: bb29cc5863b54269ae4fd08d71a42f51
I20260806 02:11:43.351709 18107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:43.368237 18107 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:43.368549 18107 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:43.377121 18107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73ba1c258af04efa9f20a874bd71a9aa: Generated new TSK 0
I20260806 02:11:43.377740 18107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:43.384289 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18017
2026-08-06T02:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:43 dist-test-slave-gp6n krb5kdc[18119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:43 dist-test-slave-gp6n krb5kdc[18119](info): set up 2 sockets
Aug 06 02:11:43 dist-test-slave-gp6n krb5kdc[18119](info): commencing operation
krb5kdc: starting...
W20260806 02:11:44.462004 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:44.590013 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:44.700858 18135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:44.701105 18135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:44.701145 18135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:44.705093 18135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:44.705154 18135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:44.705174 18135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:44.705190 18135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:44.705205 18135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:44.710355 18135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:44.711381 18135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:44.712394 18135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:44.720733 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:44.720754 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:44.720754 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:44.721099 18135 server_base.cc:1034] running on GCE node
I20260806 02:11:44.721632 18135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:44.722780 18135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:44.723991 18135 hybrid_clock.cc:648] HybridClock initialized: now 1785982304723956 us; error 48 us; skew 500 ppm
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:44.726923 18135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:44.727795 18135 webserver.cc:460] Webserver started at http://127.0.0.1:42587/ using document root <none> and password file <none>
I20260806 02:11:44.728344 18135 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:44.728446 18135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:44.728672 18135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:44.730751 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "ba152b26799b4b1d8396a4a9b759f12c"
format_stamp: "Formatted at 2026-08-06 02:11:44 on dist-test-slave-gp6n"
server_key: "4ddebf28ee2dc582e60bda824e5964f4"
server_key_iv: "ab162071241047aca6ca0fe0984dc638"
server_key_version: "encryptionkey@0"
I20260806 02:11:44.731316 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "ba152b26799b4b1d8396a4a9b759f12c"
format_stamp: "Formatted at 2026-08-06 02:11:44 on dist-test-slave-gp6n"
server_key: "4ddebf28ee2dc582e60bda824e5964f4"
server_key_iv: "ab162071241047aca6ca0fe0984dc638"
server_key_version: "encryptionkey@0"
I20260806 02:11:44.734951 18135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:11:44.737279 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:44.738543 18135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:44.738672 18135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "ba152b26799b4b1d8396a4a9b759f12c"
format_stamp: "Formatted at 2026-08-06 02:11:44 on dist-test-slave-gp6n"
server_key: "4ddebf28ee2dc582e60bda824e5964f4"
server_key_iv: "ab162071241047aca6ca0fe0984dc638"
server_key_version: "encryptionkey@0"
I20260806 02:11:44.738776 18135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:44.751533 18135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:44.754751 18135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:44.754976 18135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:44.762643 18135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44941
I20260806 02:11:44.762668 18204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44941 every 8 connection(s)
I20260806 02:11:44.763621 18135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:44.766296 18205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:44.766999 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18135
I20260806 02:11:44.767164 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:44.767489 28756 external_mini_cluster.cc:1428] Setting key 67f49502c407efa8cc21f0a864734ede
I20260806 02:11:44.772394 18205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c: Bootstrap starting.
I20260806 02:11:44.774627 18205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:44.775350 18205 log.cc:824] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:11:44.777555 18205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c: No bootstrap required, opened a new log
I20260806 02:11:44.781209 18205 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260806 02:11:44.781438 18205 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:44.781497 18205 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba152b26799b4b1d8396a4a9b759f12c, State: Initialized, Role: FOLLOWER
I20260806 02:11:44.781985 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:44.768856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53917 (local address 127.0.0.1:44941)
0806 02:11:44.769339 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:44.769356 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:44.769392 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:11:44.770038 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:44.770055 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:44.770477 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:44.770805 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:44.771606 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:44.772620 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:44.773486 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:44.773958 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:44.774133 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:44.776868 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:44.776874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:44.776896 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:44.776923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:44.778967 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:44.779448 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:44.779456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:44.779467 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:44.779584 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:44.779891 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:44.779895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:44.779897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:44.780304 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:44.780528 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:44.780798 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:11:44.781098 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260806 02:11:44.781947 18205 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260806 02:11:44.782156 18205 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:44.782258 18205 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:44.782390 18205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:44.783583 18205 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260806 02:11:44.783954 18205 leader_election.cc:302] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba152b26799b4b1d8396a4a9b759f12c; no voters:
I20260806 02:11:44.784323 18205 leader_election.cc:288] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:44.784525 18211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:44.784813 18211 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [term 1 LEADER]: Becoming Leader. State: Replica: ba152b26799b4b1d8396a4a9b759f12c, State: Running, Role: LEADER
I20260806 02:11:44.785219 18211 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260806 02:11:44.785290 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:44.785686 18205 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:44.787961 18213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba152b26799b4b1d8396a4a9b759f12c. Latest consensus state: current_term: 1 leader_uuid: "ba152b26799b4b1d8396a4a9b759f12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260806 02:11:44.788120 18213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:44.788589 18231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:44.788728 18212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba152b26799b4b1d8396a4a9b759f12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba152b26799b4b1d8396a4a9b759f12c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260806 02:11:44.790499 18212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:44.792199 18231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:44.798358 18231 catalog_manager.cc:1347] Generated new cluster ID: 16f67ec836f54c759f7182f7f4cc916f
I20260806 02:11:44.798517 18231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:44.799463 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:44.788234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57495 (local address 127.0.0.1:44941)
0806 02:11:44.788328 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:44.788332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:44.789084 (+ 752us) server_negotiation.cc:407] Connection header received
0806 02:11:44.789196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:44.789198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:44.789248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:44.789327 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:44.791513 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:44.792269 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:44.793382 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:44.793766 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:44.793879 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:44.795415 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:44.795419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:44.795422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:44.795446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:44.797170 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:44.797692 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:44.797696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:44.797697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:44.797791 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:44.798125 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:44.798129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:44.798131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:44.798354 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:44.799096 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:44.799176 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:11:44.799295 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 02:11:44.815670 18231 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:44.816910 18231 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:44.829979 18231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba152b26799b4b1d8396a4a9b759f12c: Generated new TSK 0
I20260806 02:11:44.830804 18231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:44.836723 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18135
2026-08-06T02:11:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18238](info): set up 2 sockets
Aug 06 02:11:44 dist-test-slave-gp6n krb5kdc[18238](info): commencing operation
krb5kdc: starting...
W20260806 02:11:45.941239 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:46 dist-test-slave-gp6n krb5kdc[18238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:46.070242 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:46.184894 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:46.185160 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:46.185201 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:46.189209 18252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:46.189270 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:46.189304 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:46.189322 18252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:46.189337 18252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:46.194425 18252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:46.195534 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:46.196496 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:46.205570 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:46.205570 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:46.205593 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:46.205878 18252 server_base.cc:1034] running on GCE node
I20260806 02:11:46.206529 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:46.207629 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:46.208813 18252 hybrid_clock.cc:648] HybridClock initialized: now 1785982306208791 us; error 48 us; skew 500 ppm
Aug 06 02:11:46 dist-test-slave-gp6n krb5kdc[18238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:46.211666 18252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:46.212592 18252 webserver.cc:460] Webserver started at http://127.0.0.1:43713/ using document root <none> and password file <none>
I20260806 02:11:46.213160 18252 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:46.213253 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:46.213485 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:46.215605 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "f4ead07e6dd64cbc9bfce32e3b166912"
format_stamp: "Formatted at 2026-08-06 02:11:46 on dist-test-slave-gp6n"
server_key: "54a6064d115bf7eac7be35103a4e9639"
server_key_iv: "a7eddfbd9ec7abab12844b2d9eaa5dd2"
server_key_version: "encryptionkey@0"
I20260806 02:11:46.216331 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "f4ead07e6dd64cbc9bfce32e3b166912"
format_stamp: "Formatted at 2026-08-06 02:11:46 on dist-test-slave-gp6n"
server_key: "54a6064d115bf7eac7be35103a4e9639"
server_key_iv: "a7eddfbd9ec7abab12844b2d9eaa5dd2"
server_key_version: "encryptionkey@0"
I20260806 02:11:46.221798 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:11:46.225131 18268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:46.226819 18252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:11:46.226970 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "f4ead07e6dd64cbc9bfce32e3b166912"
format_stamp: "Formatted at 2026-08-06 02:11:46 on dist-test-slave-gp6n"
server_key: "54a6064d115bf7eac7be35103a4e9639"
server_key_iv: "a7eddfbd9ec7abab12844b2d9eaa5dd2"
server_key_version: "encryptionkey@0"
I20260806 02:11:46.227088 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:46.239912 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:46.242933 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:46.243144 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:46.250600 18252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45497
I20260806 02:11:46.250671 18323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45497 every 8 connection(s)
I20260806 02:11:46.251612 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:46.254293 18324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:46.256625 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18252
I20260806 02:11:46.256789 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:46.257121 28756 external_mini_cluster.cc:1428] Setting key 7e8c2c673b71ddc0ed941f3a1064bc13
I20260806 02:11:46.260454 18324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912: Bootstrap starting.
I20260806 02:11:46.263027 18324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:46.263864 18324 log.cc:824] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:46.266005 18324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912: No bootstrap required, opened a new log
I20260806 02:11:46.267503 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:46.258563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48649 (local address 127.0.0.1:45497)
0806 02:11:46.258974 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:46.258992 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:46.259047 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:11:46.259853 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:46.259876 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:46.260291 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:46.260646 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:46.261052 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:46.262555 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:46.263522 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:46.264232 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:46.264371 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:46.265431 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:46.265437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:46.265452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:46.265950 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:46.266033 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:46.266068 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:46.266161 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:46.266370 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 02:11:46.266603 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260806 02:11:46.269644 18324 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260806 02:11:46.269922 18324 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:46.270020 18324 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ead07e6dd64cbc9bfce32e3b166912, State: Initialized, Role: FOLLOWER
I20260806 02:11:46.270498 18324 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260806 02:11:46.270668 18324 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:46.270743 18324 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:46.270807 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:46.270840 18324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:46.271680 18324 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260806 02:11:46.272070 18324 leader_election.cc:302] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ead07e6dd64cbc9bfce32e3b166912; no voters:
I20260806 02:11:46.272419 18324 leader_election.cc:288] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:46.272482 18333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:46.272722 18333 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [term 1 LEADER]: Becoming Leader. State: Replica: f4ead07e6dd64cbc9bfce32e3b166912, State: Running, Role: LEADER
I20260806 02:11:46.273082 18333 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } }
I20260806 02:11:46.273686 18324 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:46.274791 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260806 02:11:46.274923 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:46.275288 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ead07e6dd64cbc9bfce32e3b166912. Latest consensus state: current_term: 1 leader_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ead07e6dd64cbc9bfce32e3b166912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45497 } } }
I20260806 02:11:46.275911 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:46.276243 18344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:46.277779 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:46.272853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40179 (local address 127.0.0.1:45497)
0806 02:11:46.272947 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:46.272951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:46.273397 (+ 446us) server_negotiation.cc:407] Connection header received
0806 02:11:46.273550 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:46.273553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:46.273593 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:46.273679 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:46.273984 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:46.274766 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:46.275935 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:46.276272 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:46.276366 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:46.277356 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:46.277359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:46.277361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:46.277372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:46.277396 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:46.277407 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:46.277470 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:46.277595 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 02:11:46.277636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 02:11:46.279750 18344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:46.285667 18344 catalog_manager.cc:1347] Generated new cluster ID: e79c5737b1694234956c32701c0ace23
I20260806 02:11:46.285789 18344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:46.299064 18344 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:46.300436 18344 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:46.308424 18344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ead07e6dd64cbc9bfce32e3b166912: Generated new TSK 0
I20260806 02:11:46.309115 18344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:46.317687 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18252
2026-08-06T02:11:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:46 dist-test-slave-gp6n krb5kdc[18356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:46 dist-test-slave-gp6n krb5kdc[18356](info): set up 2 sockets
Aug 06 02:11:46 dist-test-slave-gp6n krb5kdc[18356](info): commencing operation
krb5kdc: starting...
W20260806 02:11:47.384308 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:47 dist-test-slave-gp6n krb5kdc[18356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:47.509920 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:47.618516 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:47.618786 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:47.618829 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:47.622830 18369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:47.622889 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:47.622910 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:47.622925 18369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:47.622972 18369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:47.628177 18369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:47.629251 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:47.630312 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:47.638793 18375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:47.638783 18377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:47.638793 18374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:47.639086 18369 server_base.cc:1034] running on GCE node
I20260806 02:11:47.639806 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:47.641022 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:47.642253 18369 hybrid_clock.cc:648] HybridClock initialized: now 1785982307642231 us; error 33 us; skew 500 ppm
Aug 06 02:11:47 dist-test-slave-gp6n krb5kdc[18356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:47.649262 18369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:47.650230 18369 webserver.cc:460] Webserver started at http://127.0.0.1:40617/ using document root <none> and password file <none>
I20260806 02:11:47.650869 18369 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:47.650967 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:47.651204 18369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:47.653258 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "5cf62ee8abaa42a781e94613f980c744"
format_stamp: "Formatted at 2026-08-06 02:11:47 on dist-test-slave-gp6n"
server_key: "053f360536f746f4bd82e400408bc3a7"
server_key_iv: "28ba8abb73956e36c16b637cb9409470"
server_key_version: "encryptionkey@0"
I20260806 02:11:47.653839 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "5cf62ee8abaa42a781e94613f980c744"
format_stamp: "Formatted at 2026-08-06 02:11:47 on dist-test-slave-gp6n"
server_key: "053f360536f746f4bd82e400408bc3a7"
server_key_iv: "28ba8abb73956e36c16b637cb9409470"
server_key_version: "encryptionkey@0"
I20260806 02:11:47.657618 18369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:11:47.660022 18384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:47.661226 18369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:47.661366 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "5cf62ee8abaa42a781e94613f980c744"
format_stamp: "Formatted at 2026-08-06 02:11:47 on dist-test-slave-gp6n"
server_key: "053f360536f746f4bd82e400408bc3a7"
server_key_iv: "28ba8abb73956e36c16b637cb9409470"
server_key_version: "encryptionkey@0"
I20260806 02:11:47.661509 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:47.665999 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:47.668802 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:47.668995 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:47.675730 18369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34097
I20260806 02:11:47.675755 18441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34097 every 8 connection(s)
I20260806 02:11:47.676615 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:47.679255 18442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:47.684522 18442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744: Bootstrap starting.
I20260806 02:11:47.686375 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18369
I20260806 02:11:47.686599 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:47.686918 28756 external_mini_cluster.cc:1428] Setting key 2f151c2f1cdd6cde97a8ce2a6aa1e98d
I20260806 02:11:47.687036 18442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:47.687954 18442 log.cc:824] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:47.689796 18442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744: No bootstrap required, opened a new log
I20260806 02:11:47.692124 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:47.688081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43219 (local address 127.0.0.1:34097)
0806 02:11:47.688615 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:47.688630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:47.688666 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:11:47.689346 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:47.689363 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:47.689639 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:47.690020 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:47.690883 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:47.690887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:47.690900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:47.690910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:47.690963 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:47.690983 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:47.691075 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:47.691216 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 02:11:47.691400 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":25856,"thread_start_us":194,"threads_started":1}
I20260806 02:11:47.693634 18442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260806 02:11:47.693862 18442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:47.693949 18442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf62ee8abaa42a781e94613f980c744, State: Initialized, Role: FOLLOWER
I20260806 02:11:47.694447 18442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260806 02:11:47.694648 18442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:47.694717 18442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:47.694813 18442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:47.694931 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:47.695891 18442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260806 02:11:47.696244 18442 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf62ee8abaa42a781e94613f980c744; no voters:
I20260806 02:11:47.696602 18442 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:47.696653 18453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:47.696986 18453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf62ee8abaa42a781e94613f980c744, State: Running, Role: LEADER
I20260806 02:11:47.697350 18453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260806 02:11:47.697755 18442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:47.698976 18455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf62ee8abaa42a781e94613f980c744. Latest consensus state: current_term: 1 leader_uuid: "5cf62ee8abaa42a781e94613f980c744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260806 02:11:47.699105 18455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:47.699432 18454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf62ee8abaa42a781e94613f980c744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf62ee8abaa42a781e94613f980c744" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260806 02:11:47.699533 18454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:47.699672 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:47.697201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35839 (local address 127.0.0.1:34097)
0806 02:11:47.697293 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:47.697297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:47.697805 (+ 508us) server_negotiation.cc:407] Connection header received
0806 02:11:47.698042 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:47.698045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:47.698078 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:47.698174 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:47.699038 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:47.699042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:47.699045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:47.699058 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:47.699088 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:47.699103 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:47.699175 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:47.699441 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 02:11:47.699483 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:11:47.700783 18464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:47.703509 18464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:47.708715 18464 catalog_manager.cc:1347] Generated new cluster ID: 5bb2510269fd4127aee31b49e4874a84
I20260806 02:11:47.708777 18464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:47.733821 18464 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:47.734175 18464 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:47.743659 18464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf62ee8abaa42a781e94613f980c744: Generated new TSK 0
I20260806 02:11:47.744525 18464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:47.754809 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18369
2026-08-06T02:11:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:47 dist-test-slave-gp6n krb5kdc[18476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:47 dist-test-slave-gp6n krb5kdc[18476](info): set up 2 sockets
Aug 06 02:11:47 dist-test-slave-gp6n krb5kdc[18476](info): commencing operation
krb5kdc: starting...
W20260806 02:11:48.821395 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:48 dist-test-slave-gp6n krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:48.945349 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:49.054762 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:49.055049 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:49.055123 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:49.059010 18490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:49.059094 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:49.059134 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:49.059193 18490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:49.059232 18490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:49.064333 18490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:49.065363 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:49.066365 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:49.074904 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:49.074944 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:49.074931 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:49.075150 18490 server_base.cc:1034] running on GCE node
I20260806 02:11:49.075811 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:49.076949 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:49.078137 18490 hybrid_clock.cc:648] HybridClock initialized: now 1785982309078121 us; error 32 us; skew 500 ppm
Aug 06 02:11:49 dist-test-slave-gp6n krb5kdc[18476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:49.080932 18490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:49.081825 18490 webserver.cc:460] Webserver started at http://127.0.0.1:38641/ using document root <none> and password file <none>
I20260806 02:11:49.082371 18490 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:49.082538 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:49.082775 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:49.084775 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "c0f74e3370e94f808d4b6cd6e86fa537"
format_stamp: "Formatted at 2026-08-06 02:11:49 on dist-test-slave-gp6n"
server_key: "a000484a84d307a726dbfde805438a3e"
server_key_iv: "ac92d7c2a32d94ac58e09e2c23aa05a1"
server_key_version: "encryptionkey@0"
I20260806 02:11:49.085351 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "c0f74e3370e94f808d4b6cd6e86fa537"
format_stamp: "Formatted at 2026-08-06 02:11:49 on dist-test-slave-gp6n"
server_key: "a000484a84d307a726dbfde805438a3e"
server_key_iv: "ac92d7c2a32d94ac58e09e2c23aa05a1"
server_key_version: "encryptionkey@0"
I20260806 02:11:49.089046 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:11:49.091615 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:49.092854 18490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:11:49.092998 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "c0f74e3370e94f808d4b6cd6e86fa537"
format_stamp: "Formatted at 2026-08-06 02:11:49 on dist-test-slave-gp6n"
server_key: "a000484a84d307a726dbfde805438a3e"
server_key_iv: "ac92d7c2a32d94ac58e09e2c23aa05a1"
server_key_version: "encryptionkey@0"
I20260806 02:11:49.093114 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:49.140311 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:49.143558 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:49.143769 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:49.150969 18490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45389
I20260806 02:11:49.150995 18561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45389 every 8 connection(s)
I20260806 02:11:49.151973 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:49.154850 18562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:49.160275 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: Bootstrap starting.
I20260806 02:11:49.162034 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18490
I20260806 02:11:49.162178 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:49.162492 28756 external_mini_cluster.cc:1428] Setting key 8a2a6260aef92d8d0cf1d7c22f69a014
I20260806 02:11:49.163066 18562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:49.164021 18562 log.cc:824] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:49.166183 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: No bootstrap required, opened a new log
I20260806 02:11:49.170535 18562 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } }
I20260806 02:11:49.170926 18562 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:49.171025 18562 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f74e3370e94f808d4b6cd6e86fa537, State: Initialized, Role: FOLLOWER
I20260806 02:11:49.171542 18562 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } }
I20260806 02:11:49.171716 18562 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:49.171795 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:49.171895 18562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:11:49 dist-test-slave-gp6n krb5kdc[18476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:11:49.173193 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } }
I20260806 02:11:49.173552 18562 leader_election.cc:302] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f74e3370e94f808d4b6cd6e86fa537; no voters:
I20260806 02:11:49.173874 18562 leader_election.cc:288] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:49.175180 18562 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:49.178864 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:49.163724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:35725 (local address 127.0.0.1:45389)
0806 02:11:49.164127 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:49.164151 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:49.164197 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:11:49.165051 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:49.165072 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:49.165405 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:49.165684 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:49.166653 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:49.167643 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:49.168933 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:49.169413 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:49.169537 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:49.172668 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:49.172674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:49.172692 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:49.172726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:49.174999 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:49.175712 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:49.175718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:49.175724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:49.175861 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:49.176225 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:49.176228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:49.176230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:49.176722 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:49.176917 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:49.177640 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 02:11:49.177861 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260806 02:11:49.180365 18567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:49.182029 18567 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [term 1 LEADER]: Becoming Leader. State: Replica: c0f74e3370e94f808d4b6cd6e86fa537, State: Running, Role: LEADER
I20260806 02:11:49.182356 18567 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } }
I20260806 02:11:49.183041 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:49.183625 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } } }
I20260806 02:11:49.183737 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:49.183913 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f74e3370e94f808d4b6cd6e86fa537. Latest consensus state: current_term: 1 leader_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f74e3370e94f808d4b6cd6e86fa537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45389 } } }
I20260806 02:11:49.183996 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537 [sys.catalog]: This master's current role is: LEADER
W20260806 02:11:49.184814 18579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:49.184894 18579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:49.185271 18583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:49.186164 18583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:49.192952 18583 catalog_manager.cc:1347] Generated new cluster ID: 0866064945f5464ba672432d8a9d3671
I20260806 02:11:49.193017 18583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:49.194612 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:49.185397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:51773 (local address 127.0.0.1:45389)
0806 02:11:49.185500 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:49.185504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:49.186057 (+ 553us) server_negotiation.cc:407] Connection header received
0806 02:11:49.186193 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:49.186198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:49.186251 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:49.186321 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:49.187032 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:49.187559 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:49.188590 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:49.188975 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:49.189095 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:49.190922 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:49.190927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:49.190932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:49.190958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:49.192742 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:49.193284 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:49.193294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:49.193296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:49.193373 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:49.193711 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:49.193714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:49.193716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:49.193888 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:49.194049 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:49.194316 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:11:49.194446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:11:49.201143 18583 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:49.202389 18583 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:49.214577 18583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f74e3370e94f808d4b6cd6e86fa537: Generated new TSK 0
I20260806 02:11:49.215286 18583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:49.224670 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18490
W20260806 02:11:49.231379 18474 connection.cc:535] client connection to 127.0.0.1:45389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45389): Connection reset by peer (error 104)
2026-08-06T02:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:49 dist-test-slave-gp6n krb5kdc[18594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:49 dist-test-slave-gp6n krb5kdc[18594](info): set up 2 sockets
Aug 06 02:11:49 dist-test-slave-gp6n krb5kdc[18594](info): commencing operation
krb5kdc: starting...
W20260806 02:11:50.297937 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:50 dist-test-slave-gp6n krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:50.427573 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:50.538532 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:50.538838 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:50.538916 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:50.542860 18611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:50.542951 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:50.542991 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:50.543028 18611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:50.543059 18611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:50.548182 18611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:50.549274 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:50.550315 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:50.559412 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:50.559449 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:50.559422 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:50.559773 18611 server_base.cc:1034] running on GCE node
I20260806 02:11:50.560343 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:50.561455 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:50.562669 18611 hybrid_clock.cc:648] HybridClock initialized: now 1785982310562660 us; error 35 us; skew 500 ppm
Aug 06 02:11:50 dist-test-slave-gp6n krb5kdc[18594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:50.565541 18611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:50.566529 18611 webserver.cc:460] Webserver started at http://127.0.0.1:40525/ using document root <none> and password file <none>
I20260806 02:11:50.567091 18611 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:50.567184 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:50.567416 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:50.569490 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "25177273720b4dc38f1e1fee2198512f"
format_stamp: "Formatted at 2026-08-06 02:11:50 on dist-test-slave-gp6n"
server_key: "a892d5474d93819b70ceb86e3625a5e8"
server_key_iv: "8f99be1c8e6ed80b18f7a338915c48b6"
server_key_version: "encryptionkey@0"
I20260806 02:11:50.570065 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "25177273720b4dc38f1e1fee2198512f"
format_stamp: "Formatted at 2026-08-06 02:11:50 on dist-test-slave-gp6n"
server_key: "a892d5474d93819b70ceb86e3625a5e8"
server_key_iv: "8f99be1c8e6ed80b18f7a338915c48b6"
server_key_version: "encryptionkey@0"
I20260806 02:11:50.573939 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:11:50.576352 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:50.577582 18611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:50.577726 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "25177273720b4dc38f1e1fee2198512f"
format_stamp: "Formatted at 2026-08-06 02:11:50 on dist-test-slave-gp6n"
server_key: "a892d5474d93819b70ceb86e3625a5e8"
server_key_iv: "8f99be1c8e6ed80b18f7a338915c48b6"
server_key_version: "encryptionkey@0"
I20260806 02:11:50.577834 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:50.616915 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:50.620186 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:50.620405 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:50.627969 18611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37765
I20260806 02:11:50.628000 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37765 every 8 connection(s)
I20260806 02:11:50.628964 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:50.631916 18682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:50.634478 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18611
I20260806 02:11:50.634642 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:50.634920 28756 external_mini_cluster.cc:1428] Setting key 82b8ff6d67b9abb15ae492441c0f8fc2
I20260806 02:11:50.638219 18682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: Bootstrap starting.
I20260806 02:11:50.640985 18682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:50.641852 18682 log.cc:824] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:50.644264 18682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: No bootstrap required, opened a new log
I20260806 02:11:50.645264 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:50.636363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:60261 (local address 127.0.0.1:37765)
0806 02:11:50.636802 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:50.636821 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:50.636866 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:11:50.637703 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:50.637726 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:50.638165 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:50.638562 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:50.638987 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:50.640479 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:50.641428 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:50.642137 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:50.642308 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:50.643314 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:50.643320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:50.643336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:50.643824 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:50.643895 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:50.643937 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:50.644075 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:50.644194 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:11:50.644331 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":124,"threads_started":1}
I20260806 02:11:50.648224 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:50.648573 18682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } }
I20260806 02:11:50.648782 18682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:50.648854 18682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25177273720b4dc38f1e1fee2198512f, State: Initialized, Role: FOLLOWER
I20260806 02:11:50.649323 18682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } }
I20260806 02:11:50.649483 18682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:50.649578 18682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:50.649688 18682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:50.650785 18682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } }
I20260806 02:11:50.651168 18682 leader_election.cc:302] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25177273720b4dc38f1e1fee2198512f; no voters:
I20260806 02:11:50.651489 18682 leader_election.cc:288] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:50.651602 18693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:50.651821 18693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [term 1 LEADER]: Becoming Leader. State: Replica: 25177273720b4dc38f1e1fee2198512f, State: Running, Role: LEADER
I20260806 02:11:50.652130 18693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } }
I20260806 02:11:50.652771 18682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:50.653249 18693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25177273720b4dc38f1e1fee2198512f. Latest consensus state: current_term: 1 leader_uuid: "25177273720b4dc38f1e1fee2198512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } } }
I20260806 02:11:50.653355 18693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:50.654042 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25177273720b4dc38f1e1fee2198512f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25177273720b4dc38f1e1fee2198512f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37765 } } }
I20260806 02:11:50.654147 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:50.655041 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:50.650343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:48061 (local address 127.0.0.1:37765)
0806 02:11:50.650485 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:50.650489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:50.650659 (+ 170us) server_negotiation.cc:407] Connection header received
0806 02:11:50.650787 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:50.650789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:50.650827 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:50.650916 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:50.651224 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:50.651969 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:50.653088 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:50.653418 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:50.653533 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:50.654580 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:50.654585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:50.654589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:50.654603 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:50.654630 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:50.654646 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:50.654738 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:50.654862 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 02:11:50.654895 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:11:50.657944 18708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:50.658008 18708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:50.658087 18697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:50.658838 18697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:50.664086 18697 catalog_manager.cc:1347] Generated new cluster ID: 4e52ad108b84499182d1afdc860d5240
I20260806 02:11:50.664148 18697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:50.681166 18697 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:50.682542 18697 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:50.695165 18697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25177273720b4dc38f1e1fee2198512f: Generated new TSK 0
I20260806 02:11:50.695935 18697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:50.700377 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18611
W20260806 02:11:50.706524 18591 connection.cc:535] client connection to 127.0.0.1:37765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37765): Connection reset by peer (error 104)
2026-08-06T02:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:50 dist-test-slave-gp6n krb5kdc[18716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:50 dist-test-slave-gp6n krb5kdc[18716](info): set up 2 sockets
Aug 06 02:11:50 dist-test-slave-gp6n krb5kdc[18716](info): commencing operation
krb5kdc: starting...
W20260806 02:11:51.769709 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:51 dist-test-slave-gp6n krb5kdc[18716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:51.897222 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:52.007628 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:52.007936 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:52.008026 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:52.012130 18730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:52.012223 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:52.012261 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:52.012298 18730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:52.012329 18730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:52.017524 18730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:52.018688 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:52.019749 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:11:52.028769 18730 server_base.cc:1034] running on GCE node
W20260806 02:11:52.028735 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:52.028728 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:52.028728 18735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:52.029596 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:52.030761 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:52.031935 18730 hybrid_clock.cc:648] HybridClock initialized: now 1785982312031916 us; error 30 us; skew 500 ppm
Aug 06 02:11:52 dist-test-slave-gp6n krb5kdc[18716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:52.035054 18730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:52.036010 18730 webserver.cc:460] Webserver started at http://127.0.0.1:35523/ using document root <none> and password file <none>
I20260806 02:11:52.036580 18730 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:52.036643 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:52.036878 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:52.039026 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "96070dfed53c420488540d4681e94902"
format_stamp: "Formatted at 2026-08-06 02:11:52 on dist-test-slave-gp6n"
server_key: "6eb172beffade8d84babf41502fb4c13"
server_key_iv: "e636766071829eb064f4e4be1045ef2f"
server_key_version: "encryptionkey@0"
I20260806 02:11:52.039613 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "96070dfed53c420488540d4681e94902"
format_stamp: "Formatted at 2026-08-06 02:11:52 on dist-test-slave-gp6n"
server_key: "6eb172beffade8d84babf41502fb4c13"
server_key_iv: "e636766071829eb064f4e4be1045ef2f"
server_key_version: "encryptionkey@0"
I20260806 02:11:52.043598 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:11:52.046046 18746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:52.047376 18730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:11:52.047540 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "96070dfed53c420488540d4681e94902"
format_stamp: "Formatted at 2026-08-06 02:11:52 on dist-test-slave-gp6n"
server_key: "6eb172beffade8d84babf41502fb4c13"
server_key_iv: "e636766071829eb064f4e4be1045ef2f"
server_key_version: "encryptionkey@0"
I20260806 02:11:52.047658 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:52.052336 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:52.055750 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:52.055946 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:52.062740 18730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45137
I20260806 02:11:52.062790 18800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45137 every 8 connection(s)
I20260806 02:11:52.063736 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:52.066365 18801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:52.071686 18801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902: Bootstrap starting.
I20260806 02:11:52.073894 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18730
I20260806 02:11:52.074059 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:52.074095 18801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:52.074369 28756 external_mini_cluster.cc:1428] Setting key 449b5894d587c2f26181de3f28d16639
I20260806 02:11:52.074967 18801 log.cc:824] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:52.077138 18801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902: No bootstrap required, opened a new log
I20260806 02:11:52.079649 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:52.075543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:38915 (local address 127.0.0.1:45137)
0806 02:11:52.075959 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:52.075975 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:52.076017 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:11:52.076821 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:52.076844 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:52.077172 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:52.077536 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:52.078217 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:52.078223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:52.078238 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:52.078249 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:52.078311 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:52.078335 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:52.078484 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:52.078587 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 02:11:52.078740 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
I20260806 02:11:52.081590 18801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260806 02:11:52.081845 18801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:52.081913 18801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96070dfed53c420488540d4681e94902, State: Initialized, Role: FOLLOWER
I20260806 02:11:52.082418 18801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260806 02:11:52.082521 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:52.082582 18801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:52.082651 18801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:52.082743 18801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:52.083618 18801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260806 02:11:52.084038 18801 leader_election.cc:302] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96070dfed53c420488540d4681e94902; no voters:
I20260806 02:11:52.084372 18801 leader_election.cc:288] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:52.084470 18811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:52.084738 18811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [term 1 LEADER]: Becoming Leader. State: Replica: 96070dfed53c420488540d4681e94902, State: Running, Role: LEADER
I20260806 02:11:52.085049 18811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260806 02:11:52.085774 18801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:52.086273 18811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96070dfed53c420488540d4681e94902. Latest consensus state: current_term: 1 leader_uuid: "96070dfed53c420488540d4681e94902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } } }
I20260806 02:11:52.086395 18811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:52.086741 18812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96070dfed53c420488540d4681e94902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96070dfed53c420488540d4681e94902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } } }
I20260806 02:11:52.086828 18812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:52.086846 18816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:52.086983 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:52.084894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:32945 (local address 127.0.0.1:45137)
0806 02:11:52.084988 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:52.084992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:52.085215 (+ 223us) server_negotiation.cc:407] Connection header received
0806 02:11:52.085381 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:52.085385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:52.085420 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:52.085511 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:52.086290 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:52.086294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:52.086296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:52.086307 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:52.086332 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:52.086343 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:52.086421 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:52.086807 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 02:11:52.086856 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:11:52.090035 18816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:52.095479 18816 catalog_manager.cc:1347] Generated new cluster ID: 69083ea3bb6f4573b4fc883593857e5a
I20260806 02:11:52.095551 18816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:52.105118 18816 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:52.105449 18816 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:52.114944 18816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96070dfed53c420488540d4681e94902: Generated new TSK 0
I20260806 02:11:52.115780 18816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:52.123589 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18730
2026-08-06T02:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:52 dist-test-slave-gp6n krb5kdc[18833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:52 dist-test-slave-gp6n krb5kdc[18833](info): set up 2 sockets
Aug 06 02:11:52 dist-test-slave-gp6n krb5kdc[18833](info): commencing operation
krb5kdc: starting...
W20260806 02:11:53.198673 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:53.330868 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:53.441052 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:53.441310 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:53.441351 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:53.445328 18848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:53.445384 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:53.445403 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:53.445420 18848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:53.445436 18848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:53.450428 18848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:53.451357 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:53.452293 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:53.461212 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:53.461212 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:53.461263 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:53.461560 18848 server_base.cc:1034] running on GCE node
I20260806 02:11:53.462116 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:53.463282 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:53.464473 18848 hybrid_clock.cc:648] HybridClock initialized: now 1785982313464442 us; error 42 us; skew 500 ppm
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:53.472014 18848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:53.473091 18848 webserver.cc:460] Webserver started at http://127.0.0.1:43353/ using document root <none> and password file <none>
I20260806 02:11:53.473695 18848 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:53.473790 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:53.474042 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:53.476240 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "2722ced5e4b64034a506499da00df2f1"
format_stamp: "Formatted at 2026-08-06 02:11:53 on dist-test-slave-gp6n"
server_key: "6051a7111866daec9e20e0235932122c"
server_key_iv: "7af37f8c9aafae480a065ecc6de3e286"
server_key_version: "encryptionkey@0"
I20260806 02:11:53.476820 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "2722ced5e4b64034a506499da00df2f1"
format_stamp: "Formatted at 2026-08-06 02:11:53 on dist-test-slave-gp6n"
server_key: "6051a7111866daec9e20e0235932122c"
server_key_iv: "7af37f8c9aafae480a065ecc6de3e286"
server_key_version: "encryptionkey@0"
I20260806 02:11:53.480523 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:11:53.482880 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:53.484089 18848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:53.484237 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "2722ced5e4b64034a506499da00df2f1"
format_stamp: "Formatted at 2026-08-06 02:11:53 on dist-test-slave-gp6n"
server_key: "6051a7111866daec9e20e0235932122c"
server_key_iv: "7af37f8c9aafae480a065ecc6de3e286"
server_key_version: "encryptionkey@0"
I20260806 02:11:53.484349 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:53.501704 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:53.504585 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:53.504803 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:53.512112 18848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41513
I20260806 02:11:53.512127 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41513 every 8 connection(s)
I20260806 02:11:53.513065 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:53.515899 18920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:53.517589 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18848
I20260806 02:11:53.517732 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:53.518028 28756 external_mini_cluster.cc:1428] Setting key 4a7b8d3b324cf0c6b40aca0973183806
I20260806 02:11:53.522374 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1: Bootstrap starting.
I20260806 02:11:53.524930 18920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:53.525835 18920 log.cc:824] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:11:53.527866 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1: No bootstrap required, opened a new log
I20260806 02:11:53.531301 18920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260806 02:11:53.531546 18920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:53.531642 18920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2722ced5e4b64034a506499da00df2f1, State: Initialized, Role: FOLLOWER
I20260806 02:11:53.532090 18920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260806 02:11:53.532244 18920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:53.532335 18920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:53.532462 18920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:53.532953 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:53.519377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:54097 (local address 127.0.0.1:41513)
0806 02:11:53.519774 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:53.519790 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:53.519824 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:11:53.520440 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:53.520456 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:53.520788 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:53.521052 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:53.521971 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:53.523074 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:53.524269 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:53.524736 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:53.524852 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:53.527712 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:53.527717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:53.527730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:53.527755 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:53.529540 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:53.530079 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:53.530085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:53.530089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:53.530202 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:53.530623 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:53.530632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:53.530633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:53.531016 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:53.531261 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:53.531572 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:11:53.532030 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260806 02:11:53.533455 18920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260806 02:11:53.533788 18920 leader_election.cc:302] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2722ced5e4b64034a506499da00df2f1; no voters:
I20260806 02:11:53.534139 18920 leader_election.cc:288] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:53.534251 18925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:53.534514 18925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2722ced5e4b64034a506499da00df2f1, State: Running, Role: LEADER
I20260806 02:11:53.534857 18925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } }
I20260806 02:11:53.535384 18920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:53.536413 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:53.536576 18926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2722ced5e4b64034a506499da00df2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } } }
I20260806 02:11:53.536693 18926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:53.537017 18927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2722ced5e4b64034a506499da00df2f1. Latest consensus state: current_term: 1 leader_uuid: "2722ced5e4b64034a506499da00df2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2722ced5e4b64034a506499da00df2f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41513 } } }
I20260806 02:11:53.537096 18927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:53.538015 18934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:53.540901 18934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:53.546967 18934 catalog_manager.cc:1347] Generated new cluster ID: 806818bf506a49398543d8ded91c39ed
I20260806 02:11:53.547101 18934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:53.548081 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:53.538778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:58261 (local address 127.0.0.1:41513)
0806 02:11:53.538877 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:53.538881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:53.539144 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:11:53.539219 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:53.539224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:53.539274 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:53.539356 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:53.540189 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:53.540906 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:53.542051 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:53.542437 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:53.542547 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:53.544140 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:53.544143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:53.544146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:53.544177 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:53.545799 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:53.546277 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:53.546280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:53.546282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:53.546364 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:53.546796 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:53.546799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:53.546801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:53.547040 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:53.547184 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:53.547843 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 02:11:53.547938 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:11:53.562527 18934 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:53.563421 18934 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:53.570237 18934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2722ced5e4b64034a506499da00df2f1: Generated new TSK 0
I20260806 02:11:53.570916 18934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:53.577535 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18848
W20260806 02:11:53.584800 18831 connection.cc:535] client connection to 127.0.0.1:41513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41513): Connection reset by peer (error 104)
2026-08-06T02:11:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18951](info): set up 2 sockets
Aug 06 02:11:53 dist-test-slave-gp6n krb5kdc[18951](info): commencing operation
krb5kdc: starting...
W20260806 02:11:54.660935 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:54 dist-test-slave-gp6n krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:54.787566 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:54.896375 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:54.896662 18964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:54.896737 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:54.900647 18964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:54.900738 18964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:54.900779 18964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:54.900838 18964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:54.900878 18964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:54.905941 18964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:54.906973 18964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:54.907965 18964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:54.916563 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:54.916592 18964 server_base.cc:1034] running on GCE node
W20260806 02:11:54.916563 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:54.916600 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:54.917358 18964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:54.918489 18964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:54.919678 18964 hybrid_clock.cc:648] HybridClock initialized: now 1785982314919661 us; error 31 us; skew 500 ppm
Aug 06 02:11:54 dist-test-slave-gp6n krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:54.922524 18964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:54.923418 18964 webserver.cc:460] Webserver started at http://127.0.0.1:45465/ using document root <none> and password file <none>
I20260806 02:11:54.923928 18964 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:54.923986 18964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:54.924147 18964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:54.926119 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "bf92521e0e404389840d4ea538a10b6b"
format_stamp: "Formatted at 2026-08-06 02:11:54 on dist-test-slave-gp6n"
server_key: "a253170b14fd66a9f9f015c654a4f36b"
server_key_iv: "ae53e01798a85def2d326b6793b4326e"
server_key_version: "encryptionkey@0"
I20260806 02:11:54.926692 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "bf92521e0e404389840d4ea538a10b6b"
format_stamp: "Formatted at 2026-08-06 02:11:54 on dist-test-slave-gp6n"
server_key: "a253170b14fd66a9f9f015c654a4f36b"
server_key_iv: "ae53e01798a85def2d326b6793b4326e"
server_key_version: "encryptionkey@0"
I20260806 02:11:54.930371 18964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:11:54.932803 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:54.934116 18964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:11:54.934250 18964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "bf92521e0e404389840d4ea538a10b6b"
format_stamp: "Formatted at 2026-08-06 02:11:54 on dist-test-slave-gp6n"
server_key: "a253170b14fd66a9f9f015c654a4f36b"
server_key_iv: "ae53e01798a85def2d326b6793b4326e"
server_key_version: "encryptionkey@0"
I20260806 02:11:54.934352 18964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:54.949499 18964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:54.952754 18964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:54.952960 18964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:54.960161 18964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34027
I20260806 02:11:54.960266 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34027 every 8 connection(s)
I20260806 02:11:54.961170 18964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:54.963744 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 18964
I20260806 02:11:54.963886 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:54.964087 19036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:54.964241 28756 external_mini_cluster.cc:1428] Setting key 88793d213ed74c83d3da3fec7e8ed941
I20260806 02:11:54.971458 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b: Bootstrap starting.
I20260806 02:11:54.973779 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:54.965575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:43015 (local address 127.0.0.1:34027)
0806 02:11:54.965988 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:54.966005 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:54.966042 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:11:54.966865 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:54.966885 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:54.967236 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:54.967544 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:54.967899 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:54.968915 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:54.970246 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:54.970809 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:54.970933 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:54.971988 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:54.971994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:54.972009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:54.972401 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:54.972458 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:54.972481 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:54.972595 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:54.972825 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:11:54.972988 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260806 02:11:54.974098 19036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:54.975051 19036 log.cc:824] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:54.977190 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:54.977378 19036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b: No bootstrap required, opened a new log
I20260806 02:11:54.981462 19036 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260806 02:11:54.981732 19036 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:54.981820 19036 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf92521e0e404389840d4ea538a10b6b, State: Initialized, Role: FOLLOWER
I20260806 02:11:54.982371 19036 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260806 02:11:54.982600 19036 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:54.982677 19036 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:54.982780 19036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:54.983489 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:54.979307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:43089 (local address 127.0.0.1:34027)
0806 02:11:54.979450 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:54.979455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:54.979648 (+ 193us) server_negotiation.cc:407] Connection header received
0806 02:11:54.979807 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:54.979815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:54.979856 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:54.979939 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:54.980226 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:54.980728 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:54.981586 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:54.981879 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:54.981994 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:54.982988 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:54.982994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:54.983001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:54.983018 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:54.983050 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:54.983066 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:54.983158 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:54.983313 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 02:11:54.983351 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:11:54.984061 19036 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260806 02:11:54.984428 19036 leader_election.cc:302] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf92521e0e404389840d4ea538a10b6b; no voters:
I20260806 02:11:54.984771 19036 leader_election.cc:288] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:54.984890 19046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:54.985147 19046 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [term 1 LEADER]: Becoming Leader. State: Replica: bf92521e0e404389840d4ea538a10b6b, State: Running, Role: LEADER
I20260806 02:11:54.985481 19046 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260806 02:11:54.986061 19036 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:54.987279 19048 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf92521e0e404389840d4ea538a10b6b. Latest consensus state: current_term: 1 leader_uuid: "bf92521e0e404389840d4ea538a10b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260806 02:11:54.987263 19047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf92521e0e404389840d4ea538a10b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf92521e0e404389840d4ea538a10b6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260806 02:11:54.987421 19048 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:54.987425 19047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:54.987839 19056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:54.991122 19056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:54.996894 19056 catalog_manager.cc:1347] Generated new cluster ID: a1bb66f9abda4420bbf36bee73835fb7
I20260806 02:11:54.996971 19056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:55.006138 19056 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:55.007105 19056 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:55.012876 19056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf92521e0e404389840d4ea538a10b6b: Generated new TSK 0
I20260806 02:11:55.013525 19056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:55.020533 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 18964
W20260806 02:11:55.026903 18949 connection.cc:535] client connection to 127.0.0.1:34027 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34027): Connection reset by peer (error 104)
2026-08-06T02:11:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:55 dist-test-slave-gp6n krb5kdc[19068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:55 dist-test-slave-gp6n krb5kdc[19068](info): set up 2 sockets
Aug 06 02:11:55 dist-test-slave-gp6n krb5kdc[19068](info): commencing operation
krb5kdc: starting...
W20260806 02:11:56.116842 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:56 dist-test-slave-gp6n krb5kdc[19068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:11:56.249836 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:56.358944 19083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:56.359226 19083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:56.359268 19083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:56.363246 19083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:56.363312 19083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:56.363332 19083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:56.363348 19083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:56.363363 19083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:56.368619 19083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:56.369655 19083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:56.370659 19083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:56.379339 19089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:56.379370 19088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:56.379390 19091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:56.379654 19083 server_base.cc:1034] running on GCE node
I20260806 02:11:56.380220 19083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:56.381243 19083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:56.382443 19083 hybrid_clock.cc:648] HybridClock initialized: now 1785982316382388 us; error 29 us; skew 500 ppm
Aug 06 02:11:56 dist-test-slave-gp6n krb5kdc[19068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:56.385299 19083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:11:56.386247 19083 webserver.cc:460] Webserver started at http://127.0.0.1:33283/ using document root <none> and password file <none>
I20260806 02:11:56.386847 19083 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:56.386911 19083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:56.387131 19083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:56.389149 19083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "b756c50897384a0b8442dc376ea74854"
format_stamp: "Formatted at 2026-08-06 02:11:56 on dist-test-slave-gp6n"
server_key: "b4f78dde74fe0d2447b9a83afc95d5c5"
server_key_iv: "59a26284de47fece77302bbda12046f9"
server_key_version: "encryptionkey@0"
I20260806 02:11:56.389741 19083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "b756c50897384a0b8442dc376ea74854"
format_stamp: "Formatted at 2026-08-06 02:11:56 on dist-test-slave-gp6n"
server_key: "b4f78dde74fe0d2447b9a83afc95d5c5"
server_key_iv: "59a26284de47fece77302bbda12046f9"
server_key_version: "encryptionkey@0"
I20260806 02:11:56.393451 19083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:11:56.396025 19098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:56.397212 19083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 02:11:56.397341 19083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "b756c50897384a0b8442dc376ea74854"
format_stamp: "Formatted at 2026-08-06 02:11:56 on dist-test-slave-gp6n"
server_key: "b4f78dde74fe0d2447b9a83afc95d5c5"
server_key_iv: "59a26284de47fece77302bbda12046f9"
server_key_version: "encryptionkey@0"
I20260806 02:11:56.397440 19083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:56.401980 19083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:56.404774 19083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:56.404964 19083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:56.412201 19083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34229
I20260806 02:11:56.412230 19155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34229 every 8 connection(s)
I20260806 02:11:56.413085 19083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:56.415916 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:56.416226 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19083
I20260806 02:11:56.416371 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:56.416673 28756 external_mini_cluster.cc:1428] Setting key 9edda7f45ed4270e6d938210d6bfffef
I20260806 02:11:56.422822 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:56.418187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:56321 (local address 127.0.0.1:34229)
0806 02:11:56.418618 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:56.418639 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:56.418685 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:11:56.419855 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:56.419880 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:56.420259 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:56.420651 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:56.421337 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:56.421343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:56.421357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:56.421369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:56.421430 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:56.421458 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:56.421606 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:56.421680 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:11:56.421852 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":104,"threads_started":1}
I20260806 02:11:56.423933 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854: Bootstrap starting.
I20260806 02:11:56.425455 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:56.426470 19156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:56.427367 19156 log.cc:824] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:56.429167 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:56.427441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:33645 (local address 127.0.0.1:34229)
0806 02:11:56.427570 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:56.427574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:56.427816 (+ 242us) server_negotiation.cc:407] Connection header received
0806 02:11:56.427914 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:56.427916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:56.427944 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:56.428013 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:56.428777 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:56.428781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:56.428784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:56.428805 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:56.428834 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:56.428847 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:56.428944 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:56.429004 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 02:11:56.429053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:11:56.429765 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854: No bootstrap required, opened a new log
I20260806 02:11:56.433439 19156 raft_consensus.cc:348] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260806 02:11:56.433698 19156 raft_consensus.cc:374] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:56.433785 19156 raft_consensus.cc:729] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b756c50897384a0b8442dc376ea74854, State: Initialized, Role: FOLLOWER
I20260806 02:11:56.434216 19156 consensus_queue.cc:260] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260806 02:11:56.434386 19156 raft_consensus.cc:388] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:56.434489 19156 raft_consensus.cc:482] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:56.434613 19156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:56.435534 19156 raft_consensus.cc:504] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260806 02:11:56.435890 19156 leader_election.cc:302] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b756c50897384a0b8442dc376ea74854; no voters:
I20260806 02:11:56.436203 19156 leader_election.cc:288] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:56.436319 19167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:56.436594 19167 raft_consensus.cc:686] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [term 1 LEADER]: Becoming Leader. State: Replica: b756c50897384a0b8442dc376ea74854, State: Running, Role: LEADER
I20260806 02:11:56.436880 19167 consensus_queue.cc:237] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } }
I20260806 02:11:56.437481 19156 sys_catalog.cc:564] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:56.438475 19169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b756c50897384a0b8442dc376ea74854. Latest consensus state: current_term: 1 leader_uuid: "b756c50897384a0b8442dc376ea74854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } } }
I20260806 02:11:56.438604 19169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:56.438941 19168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b756c50897384a0b8442dc376ea74854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b756c50897384a0b8442dc376ea74854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34229 } } }
I20260806 02:11:56.439019 19176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:56.439044 19168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:56.441905 19176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:56.447199 19176 catalog_manager.cc:1347] Generated new cluster ID: 54e495c9216c45e4b230b75de3d574bd
I20260806 02:11:56.447263 19176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:56.457921 19176 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:56.458237 19176 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:56.470191 19176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b756c50897384a0b8442dc376ea74854: Generated new TSK 0
I20260806 02:11:56.470914 19176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:56.475185 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19083
2026-08-06T02:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:56 dist-test-slave-gp6n krb5kdc[19188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:56 dist-test-slave-gp6n krb5kdc[19188](info): set up 2 sockets
Aug 06 02:11:56 dist-test-slave-gp6n krb5kdc[19188](info): commencing operation
krb5kdc: starting...
W20260806 02:11:57.541482 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:11:57.670219 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39793
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:57.778679 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:57.778971 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:57.779045 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:57.782984 19201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:57.783071 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:57.783109 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:57.783167 19201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:57.783207 19201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:57.788378 19201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:39793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:57.789466 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:57.790516 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:57.799280 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:57.799238 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:57.799310 19201 server_base.cc:1034] running on GCE node
W20260806 02:11:57.799238 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:57.799985 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:57.801034 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:57.802201 19201 hybrid_clock.cc:648] HybridClock initialized: now 1785982317802189 us; error 24 us; skew 500 ppm
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982317, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:57.804903 19201 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:57.805737 19201 webserver.cc:460] Webserver started at http://127.28.21.62:36421/ using document root <none> and password file <none>
I20260806 02:11:57.806291 19201 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:57.806351 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:57.806574 19201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:57.808532 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "87278be90d1d4ae8a303bcbfcd18315b"
format_stamp: "Formatted at 2026-08-06 02:11:57 on dist-test-slave-gp6n"
server_key: "1ba4ac364a404a2660d5874495776539"
server_key_iv: "787b60efc7b2bcf87abfef139f3ab97a"
server_key_version: "encryptionkey@0"
I20260806 02:11:57.809033 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "87278be90d1d4ae8a303bcbfcd18315b"
format_stamp: "Formatted at 2026-08-06 02:11:57 on dist-test-slave-gp6n"
server_key: "1ba4ac364a404a2660d5874495776539"
server_key_iv: "787b60efc7b2bcf87abfef139f3ab97a"
server_key_version: "encryptionkey@0"
I20260806 02:11:57.812579 19201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 02:11:57.814813 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:57.816038 19201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:57.816166 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "87278be90d1d4ae8a303bcbfcd18315b"
format_stamp: "Formatted at 2026-08-06 02:11:57 on dist-test-slave-gp6n"
server_key: "1ba4ac364a404a2660d5874495776539"
server_key_iv: "787b60efc7b2bcf87abfef139f3ab97a"
server_key_version: "encryptionkey@0"
I20260806 02:11:57.816262 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:57.829409 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:57.832110 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:57.832307 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:57.839155 19201 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:39793
I20260806 02:11:57.839213 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:39793 every 8 connection(s)
I20260806 02:11:57.840147 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:57.842936 19276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:57.846750 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19201
I20260806 02:11:57.846901 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:57.847172 28756 external_mini_cluster.cc:1428] Setting key 318e861c606a600c4affad6ebf5d4f13
I20260806 02:11:57.848784 19276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: Bootstrap starting.
I20260806 02:11:57.851552 19276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:57.852440 19276 log.cc:824] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:57.854528 19276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: No bootstrap required, opened a new log
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:11:57.858594 19276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } }
I20260806 02:11:57.858862 19276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:57.858934 19276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87278be90d1d4ae8a303bcbfcd18315b, State: Initialized, Role: FOLLOWER
I20260806 02:11:57.859424 19276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } }
I20260806 02:11:57.859630 19276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:57.859704 19276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:57.859805 19276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:57.860914 19276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } }
I20260806 02:11:57.861302 19276 leader_election.cc:302] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87278be90d1d4ae8a303bcbfcd18315b; no voters:
I20260806 02:11:57.861666 19276 leader_election.cc:288] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:57.861766 19281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:57.862011 19281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [term 1 LEADER]: Becoming Leader. State: Replica: 87278be90d1d4ae8a303bcbfcd18315b, State: Running, Role: LEADER
I20260806 02:11:57.862371 19281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } }
I20260806 02:11:57.862870 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:57.848589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57591 (local address 127.28.21.62:39793)
0806 02:11:57.849034 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:57.849056 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:57.849098 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:11:57.849948 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:57.849976 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:57.850464 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:57.850814 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:57.851517 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:57.853015 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:57.853914 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:57.854637 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:57.854794 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:57.857269 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:57.857275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:57.857291 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:57.857323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:57.859345 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:57.859926 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:57.859932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:57.859938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:57.860095 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:57.860406 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:57.860409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:57.860411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:57.860943 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:57.861153 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:57.861658 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 02:11:57.861920 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":107,"threads_started":1}
I20260806 02:11:57.862932 19276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:57.864877 19282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87278be90d1d4ae8a303bcbfcd18315b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } } }
I20260806 02:11:57.865288 19282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:57.865517 19283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87278be90d1d4ae8a303bcbfcd18315b. Latest consensus state: current_term: 1 leader_uuid: "87278be90d1d4ae8a303bcbfcd18315b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87278be90d1d4ae8a303bcbfcd18315b" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39793 } } }
I20260806 02:11:57.865659 19283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:57.866395 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 02:11:57.867982 19295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:11:57.868069 19295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:11:57.868135 19296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:57.868983 19296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:57.874116 19296 catalog_manager.cc:1347] Generated new cluster ID: 7b13d19808104c11bbd5bef6c17e6d3e
I20260806 02:11:57.874176 19296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:57.879719 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:57.868553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60291 (local address 127.28.21.62:39793)
0806 02:11:57.868648 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:57.868651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:57.870713 (+ 2062us) server_negotiation.cc:407] Connection header received
0806 02:11:57.870830 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:57.870833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:57.870877 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:57.870961 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:57.871770 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:57.872411 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:57.873512 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:57.873945 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:57.874069 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:57.876147 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:57.876151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:57.876154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:11:57.876179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:57.877835 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:57.878363 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:57.878366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:57.878368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:57.878490 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:11:57.878823 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:11:57.878826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:11:57.878828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:11:57.879049 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:57.879195 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:57.879453 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:11:57.879552 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 02:11:57.885138 19296 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:57.885977 19296 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:57.890729 19296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87278be90d1d4ae8a303bcbfcd18315b: Generated new TSK 0
I20260806 02:11:57.891469 19296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:57.898764 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19201
W20260806 02:11:57.906543 19186 connection.cc:535] client connection to 127.28.21.62:39793 recv error: Network error: failed to read from TLS socket (remote: 127.28.21.62:39793): Connection reset by peer (error 104)
2026-08-06T02:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19307](info): set up 2 sockets
Aug 06 02:11:57 dist-test-slave-gp6n krb5kdc[19307](info): commencing operation
krb5kdc: starting...
W20260806 02:11:58.988921 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:11:59 dist-test-slave-gp6n krb5kdc[19307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:11:59.117425 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37765
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:37765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:11:59.228719 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:11:59.228977 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:11:59.229025 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:11:59.233016 19320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:11:59.233069 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:11:59.233088 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:11:59.233105 19320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:11:59.233120 19320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:11:59.238313 19320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:37765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:11:59.239362 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:11:59.240340 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:11:59.249097 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:11:59.249055 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:59.249162 19320 server_base.cc:1034] running on GCE node
W20260806 02:11:59.249150 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:11:59.250039 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:11:59.251261 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:11:59.252442 19320 hybrid_clock.cc:648] HybridClock initialized: now 1785982319252414 us; error 39 us; skew 500 ppm
Aug 06 02:11:59 dist-test-slave-gp6n krb5kdc[19307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:11:59.260070 19320 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:11:59.260900 19320 webserver.cc:460] Webserver started at http://127.28.21.62:39973/ using document root <none> and password file <none>
I20260806 02:11:59.261404 19320 fs_manager.cc:359] Metadata directory not provided
I20260806 02:11:59.261461 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:11:59.261619 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:11:59.263617 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "2d847872949b4c848820c46153a7d0c7"
format_stamp: "Formatted at 2026-08-06 02:11:59 on dist-test-slave-gp6n"
server_key: "b46eb2d92b9ef750e66537b8297295bc"
server_key_iv: "88ed1913a29c0ba0be226129866d843e"
server_key_version: "encryptionkey@0"
I20260806 02:11:59.264200 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "2d847872949b4c848820c46153a7d0c7"
format_stamp: "Formatted at 2026-08-06 02:11:59 on dist-test-slave-gp6n"
server_key: "b46eb2d92b9ef750e66537b8297295bc"
server_key_iv: "88ed1913a29c0ba0be226129866d843e"
server_key_version: "encryptionkey@0"
I20260806 02:11:59.267854 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:11:59.270263 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:11:59.271559 19320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:11:59.271687 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "2d847872949b4c848820c46153a7d0c7"
format_stamp: "Formatted at 2026-08-06 02:11:59 on dist-test-slave-gp6n"
server_key: "b46eb2d92b9ef750e66537b8297295bc"
server_key_iv: "88ed1913a29c0ba0be226129866d843e"
server_key_version: "encryptionkey@0"
I20260806 02:11:59.271795 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:11:59.302173 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:11:59.305425 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:11:59.305632 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:11:59.312673 19320 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:37765
I20260806 02:11:59.312697 19393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:37765 every 8 connection(s)
I20260806 02:11:59.313632 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:11:59.314850 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19320
I20260806 02:11:59.314962 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:11:59.315254 28756 external_mini_cluster.cc:1428] Setting key 9e4498f301b4dd7acc4f1d920358bf96
I20260806 02:11:59.316758 19394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:11:59.323637 19394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7: Bootstrap starting.
I20260806 02:11:59.325661 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:59.316663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58039 (local address 127.28.21.62:37765)
0806 02:11:59.317097 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:59.317115 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:59.317158 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:11:59.318011 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:59.318034 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:59.318483 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:59.318850 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:59.319264 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:59.320788 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:59.321789 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:59.322535 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:59.322701 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:59.323596 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:59.323602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:59.323618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:59.324119 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:59.324185 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:59.324215 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:59.324350 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:59.324592 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:11:59.324722 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260806 02:11:59.326395 19394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7: Neither blocks nor log segments found. Creating new log.
I20260806 02:11:59.327195 19394 log.cc:824] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:11:59.328847 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:11:59.329440 19394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7: No bootstrap required, opened a new log
I20260806 02:11:59.333653 19394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } }
I20260806 02:11:59.333909 19394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:11:59.334017 19394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d847872949b4c848820c46153a7d0c7, State: Initialized, Role: FOLLOWER
I20260806 02:11:59.334563 19394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } }
I20260806 02:11:59.334734 19394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:11:59.334863 19394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:11:59.334991 19394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:11:59.335358 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:11:59.331085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32939 (local address 127.28.21.62:37765)
0806 02:11:59.331224 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:11:59.331229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:11:59.331445 (+ 216us) server_negotiation.cc:407] Connection header received
0806 02:11:59.331579 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:11:59.331581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:11:59.331618 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:11:59.331687 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:11:59.331979 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:59.332504 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:59.333627 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:11:59.333873 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:11:59.334020 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:11:59.334874 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:11:59.334878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:11:59.334880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:11:59.334893 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:11:59.334943 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:11:59.334958 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:11:59.335075 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:11:59.335217 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 02:11:59.335256 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:11:59.336198 19394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } }
I20260806 02:11:59.336575 19394 leader_election.cc:302] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d847872949b4c848820c46153a7d0c7; no voters:
I20260806 02:11:59.336886 19394 leader_election.cc:288] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:11:59.337051 19404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:11:59.337324 19404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2d847872949b4c848820c46153a7d0c7, State: Running, Role: LEADER
I20260806 02:11:59.337687 19404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } }
I20260806 02:11:59.338165 19394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:11:59.339538 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d847872949b4c848820c46153a7d0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } } }
I20260806 02:11:59.339540 19406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d847872949b4c848820c46153a7d0c7. Latest consensus state: current_term: 1 leader_uuid: "2d847872949b4c848820c46153a7d0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d847872949b4c848820c46153a7d0c7" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 37765 } } }
I20260806 02:11:59.339758 19406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:59.339705 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:11:59.340152 19416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:11:59.343091 19416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:11:59.348141 19416 catalog_manager.cc:1347] Generated new cluster ID: 4951b7eef56945c1a3051a3aa20d71f5
I20260806 02:11:59.348213 19416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:11:59.357707 19416 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:11:59.358672 19416 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:11:59.365306 19416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d847872949b4c848820c46153a7d0c7: Generated new TSK 0
I20260806 02:11:59.365993 19416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:11:59.370931 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19320
W20260806 02:11:59.377415 19305 connection.cc:535] client connection to 127.28.21.62:37765 recv error: Network error: failed to read from TLS socket (remote: 127.28.21.62:37765): Connection reset by peer (error 104)
2026-08-06T02:11:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:11:59 dist-test-slave-gp6n krb5kdc[19427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:11:59 dist-test-slave-gp6n krb5kdc[19427](info): set up 2 sockets
Aug 06 02:11:59 dist-test-slave-gp6n krb5kdc[19427](info): commencing operation
krb5kdc: starting...
W20260806 02:12:00.457188 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:00 dist-test-slave-gp6n krb5kdc[19427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:00.585712 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44127
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:44127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:00.697690 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:00.697991 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:00.698072 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:00.702107 19440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:00.702204 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:00.702245 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:00.702315 19440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:00.702378 19440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:00.707615 19440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:44127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:00.708678 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:00.709702 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:00.718042 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:00.718052 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:00.718084 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:00.718108 19440 server_base.cc:1034] running on GCE node
I20260806 02:12:00.718871 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:00.719857 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:00.721053 19440 hybrid_clock.cc:648] HybridClock initialized: now 1785982320721039 us; error 34 us; skew 500 ppm
Aug 06 02:12:00 dist-test-slave-gp6n krb5kdc[19427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:00.723821 19440 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:00.724651 19440 webserver.cc:460] Webserver started at http://127.28.21.62:34923/ using document root <none> and password file <none>
I20260806 02:12:00.725207 19440 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:00.725301 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:00.725526 19440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:00.727660 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "abbc6b858ff34f87850e1eb0ecf46f35"
format_stamp: "Formatted at 2026-08-06 02:12:00 on dist-test-slave-gp6n"
server_key: "4013cd070d8078b5b2de0650f2d594d6"
server_key_iv: "4f0ab0737274d0f7b92e43c44a37e2e7"
server_key_version: "encryptionkey@0"
I20260806 02:12:00.728233 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "abbc6b858ff34f87850e1eb0ecf46f35"
format_stamp: "Formatted at 2026-08-06 02:12:00 on dist-test-slave-gp6n"
server_key: "4013cd070d8078b5b2de0650f2d594d6"
server_key_iv: "4f0ab0737274d0f7b92e43c44a37e2e7"
server_key_version: "encryptionkey@0"
I20260806 02:12:00.731874 19440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:12:00.734285 19456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:00.735765 19440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:12:00.735908 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "abbc6b858ff34f87850e1eb0ecf46f35"
format_stamp: "Formatted at 2026-08-06 02:12:00 on dist-test-slave-gp6n"
server_key: "4013cd070d8078b5b2de0650f2d594d6"
server_key_iv: "4f0ab0737274d0f7b92e43c44a37e2e7"
server_key_version: "encryptionkey@0"
I20260806 02:12:00.736025 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:00.740630 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:00.743847 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:00.744053 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:00.751318 19440 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:44127
I20260806 02:12:00.751341 19509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:44127 every 8 connection(s)
I20260806 02:12:00.752207 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:00.752435 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19440
I20260806 02:12:00.752586 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:00.752942 28756 external_mini_cluster.cc:1428] Setting key 6a39e72d27aa529f98f42c7ad8ffbefc
I20260806 02:12:00.755525 19510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:00.758109 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:00.754442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33231 (local address 127.28.21.62:44127)
0806 02:12:00.754887 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:00.754901 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:00.754935 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:12:00.755569 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:00.755584 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:00.755899 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:00.756220 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:00.757002 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:00.757007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:00.757017 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:00.757027 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:00.757078 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:00.757097 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:00.757244 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:00.757314 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:12:00.757464 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":158,"threads_started":1}
I20260806 02:12:00.760761 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:00.761919 19510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35: Bootstrap starting.
I20260806 02:12:00.764438 19510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:00.765210 19510 log.cc:824] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:00.766179 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:00.762840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41009 (local address 127.28.21.62:44127)
0806 02:12:00.762948 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:00.762951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:00.764754 (+ 1803us) server_negotiation.cc:407] Connection header received
0806 02:12:00.764898 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:00.764900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:00.764928 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:00.765014 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:00.765747 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:00.765750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:00.765753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:00.765763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:00.765789 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:00.765807 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:00.765894 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:00.766021 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 02:12:00.766058 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:12:00.767520 19510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35: No bootstrap required, opened a new log
I20260806 02:12:00.771296 19510 raft_consensus.cc:348] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } }
I20260806 02:12:00.771533 19510 raft_consensus.cc:374] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:12:00.771572 19510 raft_consensus.cc:729] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbc6b858ff34f87850e1eb0ecf46f35, State: Initialized, Role: FOLLOWER
I20260806 02:12:00.771950 19510 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } }
I20260806 02:12:00.772070 19510 raft_consensus.cc:388] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:12:00.772120 19510 raft_consensus.cc:482] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:12:00.772197 19510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:00.773309 19510 raft_consensus.cc:504] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } }
I20260806 02:12:00.773615 19510 leader_election.cc:302] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbc6b858ff34f87850e1eb0ecf46f35; no voters:
I20260806 02:12:00.773873 19510 leader_election.cc:288] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:12:00.773993 19521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:00.774243 19521 raft_consensus.cc:686] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [term 1 LEADER]: Becoming Leader. State: Replica: abbc6b858ff34f87850e1eb0ecf46f35, State: Running, Role: LEADER
I20260806 02:12:00.774627 19521 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } }
I20260806 02:12:00.775192 19510 sys_catalog.cc:564] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:00.776230 19523 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbc6b858ff34f87850e1eb0ecf46f35. Latest consensus state: current_term: 1 leader_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } } }
I20260806 02:12:00.776504 19522 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbc6b858ff34f87850e1eb0ecf46f35" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 44127 } } }
I20260806 02:12:00.776609 19523 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:00.776643 19522 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:00.777099 19531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:00.780237 19531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:12:00.785884 19531 catalog_manager.cc:1347] Generated new cluster ID: 9648feaab06f4f1fb271225fd30a18bb
I20260806 02:12:00.785989 19531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:00.810566 19531 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:00.810922 19531 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:00.815904 19531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abbc6b858ff34f87850e1eb0ecf46f35: Generated new TSK 0
I20260806 02:12:00.816584 19531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:12:00.822297 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19440
2026-08-06T02:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:12:00 dist-test-slave-gp6n krb5kdc[19545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:12:00 dist-test-slave-gp6n krb5kdc[19545](info): set up 2 sockets
Aug 06 02:12:00 dist-test-slave-gp6n krb5kdc[19545](info): commencing operation
krb5kdc: starting...
W20260806 02:12:01.884712 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:01 dist-test-slave-gp6n krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:02.009774 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39659
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:02.119383 19559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:02.119683 19559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:02.119758 19559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:02.123720 19559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:02.123803 19559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:02.123838 19559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:02.123871 19559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:02.123911 19559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:02.129190 19559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:39659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:02.130256 19559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:02.131357 19559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:02.140051 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:02.140098 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:02.140193 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:02.140087 19559 server_base.cc:1034] running on GCE node
I20260806 02:12:02.140892 19559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:02.141981 19559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:02.143183 19559 hybrid_clock.cc:648] HybridClock initialized: now 1785982322143165 us; error 33 us; skew 500 ppm
Aug 06 02:12:02 dist-test-slave-gp6n krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982322, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:02.146137 19559 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:02.147075 19559 webserver.cc:460] Webserver started at http://127.28.21.62:46011/ using document root <none> and password file <none>
I20260806 02:12:02.147650 19559 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:02.147745 19559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:02.147953 19559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:02.150015 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "b5ce9b6882fc4fc0a58fb265ea447281"
format_stamp: "Formatted at 2026-08-06 02:12:02 on dist-test-slave-gp6n"
server_key: "51a401c9daea64c2f0b8effcac0de3cf"
server_key_iv: "21eaf99afa5badd41ecf4315136742ac"
server_key_version: "encryptionkey@0"
I20260806 02:12:02.150650 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "b5ce9b6882fc4fc0a58fb265ea447281"
format_stamp: "Formatted at 2026-08-06 02:12:02 on dist-test-slave-gp6n"
server_key: "51a401c9daea64c2f0b8effcac0de3cf"
server_key_iv: "21eaf99afa5badd41ecf4315136742ac"
server_key_version: "encryptionkey@0"
I20260806 02:12:02.154330 19559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:12:02.156772 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:02.157912 19559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:12:02.158052 19559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "b5ce9b6882fc4fc0a58fb265ea447281"
format_stamp: "Formatted at 2026-08-06 02:12:02 on dist-test-slave-gp6n"
server_key: "51a401c9daea64c2f0b8effcac0de3cf"
server_key_iv: "21eaf99afa5badd41ecf4315136742ac"
server_key_version: "encryptionkey@0"
I20260806 02:12:02.158162 19559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:02.180802 19559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:02.183835 19559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:02.184062 19559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:02.191121 19559 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:39659
I20260806 02:12:02.191139 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:39659 every 8 connection(s)
I20260806 02:12:02.192080 19559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:02.194861 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:02.196624 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19559
I20260806 02:12:02.196795 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:02.197084 28756 external_mini_cluster.cc:1428] Setting key 7b8e2be3f0c04ee8da92c5d68627c9e5
I20260806 02:12:02.201321 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281: Bootstrap starting.
I20260806 02:12:02.203779 19629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:02.204617 19629 log.cc:824] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:12:02 dist-test-slave-gp6n krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:02.206789 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281: No bootstrap required, opened a new log
I20260806 02:12:02.209606 19629 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } }
I20260806 02:12:02.209851 19629 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:12:02.209924 19629 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ce9b6882fc4fc0a58fb265ea447281, State: Initialized, Role: FOLLOWER
I20260806 02:12:02.210387 19629 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } }
I20260806 02:12:02.210587 19629 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:12:02.210654 19629 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:12:02.210747 19629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:02.211658 19629 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } }
I20260806 02:12:02.211983 19629 leader_election.cc:302] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ce9b6882fc4fc0a58fb265ea447281; no voters:
I20260806 02:12:02.212301 19629 leader_election.cc:288] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:12:02.212379 19636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:02.212531 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:02.198441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36429 (local address 127.28.21.62:39659)
0806 02:12:02.198847 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:02.198862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:02.198897 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:12:02.199529 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:02.199546 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:02.199870 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:02.200173 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:02.201160 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:02.202322 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:02.203573 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:02.204143 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:02.204270 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:02.207113 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:02.207118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:02.207131 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:02.207157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:02.209309 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:02.209779 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:02.209784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:02.209788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:02.209955 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:02.210242 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:02.210247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:02.210255 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:02.210851 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:02.211053 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:02.211340 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:12:02.211627 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
I20260806 02:12:02.212615 19636 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [term 1 LEADER]: Becoming Leader. State: Replica: b5ce9b6882fc4fc0a58fb265ea447281, State: Running, Role: LEADER
I20260806 02:12:02.212886 19636 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } }
I20260806 02:12:02.213559 19629 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:02.213967 19638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ce9b6882fc4fc0a58fb265ea447281. Latest consensus state: current_term: 1 leader_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } } }
I20260806 02:12:02.214221 19638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:02.214134 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ce9b6882fc4fc0a58fb265ea447281" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 39659 } } }
I20260806 02:12:02.214540 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:02.214637 19641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:02.215698 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:02.217890 19641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:12:02.224967 19653 mvcc.cc:205] Tried to move back new op lower bound from 7315383591819333632 to 7315383591784837120. Current Snapshot: MvccSnapshot[applied={T|T < 7315383591819333632}]
I20260806 02:12:02.226203 19641 catalog_manager.cc:1347] Generated new cluster ID: a6b949ce6303417b9938ff10e0dbab54
I20260806 02:12:02.226441 19641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:02.235736 19641 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:02.236928 19641 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:02.239140 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:02.223842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44889 (local address 127.28.21.62:39659)
0806 02:12:02.223947 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:02.223952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:02.230739 (+ 6787us) server_negotiation.cc:407] Connection header received
0806 02:12:02.230839 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:02.230843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:02.230893 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:02.230986 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:02.231722 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:02.232540 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:02.233668 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:02.234036 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:02.234134 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:02.235802 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:02.235805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:02.235808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:02.235851 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:02.237481 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:02.237940 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:02.237943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:02.237945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:02.238026 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:02.238306 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:02.238309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:02.238311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:02.238514 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:02.238627 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:02.238873 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:12:02.238990 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:12:02.250864 19641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ce9b6882fc4fc0a58fb265ea447281: Generated new TSK 0
I20260806 02:12:02.251590 19641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:12:02.261612 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19559
2026-08-06T02:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:12:02 dist-test-slave-gp6n krb5kdc[19662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:12:02 dist-test-slave-gp6n krb5kdc[19662](info): set up 2 sockets
Aug 06 02:12:02 dist-test-slave-gp6n krb5kdc[19662](info): commencing operation
krb5kdc: starting...
W20260806 02:12:03.333395 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:03 dist-test-slave-gp6n krb5kdc[19662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:03.463713 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34017
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:03.582734 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:03.582995 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:03.583036 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:03.586952 19675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:03.587010 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:03.587029 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:03.587046 19675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:03.587061 19675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:03.592273 19675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:03.593250 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:03.594246 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:03.602597 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:03.602597 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:03.602665 19675 server_base.cc:1034] running on GCE node
W20260806 02:12:03.602893 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:03.603410 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:03.604482 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:03.605679 19675 hybrid_clock.cc:648] HybridClock initialized: now 1785982323605667 us; error 25 us; skew 500 ppm
Aug 06 02:12:03 dist-test-slave-gp6n krb5kdc[19662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982323, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:03.608472 19675 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:03.609460 19675 webserver.cc:460] Webserver started at http://127.28.21.62:42989/ using document root <none> and password file <none>
I20260806 02:12:03.610015 19675 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:03.610106 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:03.610342 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:03.612408 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "5834bfb4e5a7429da3080515dd762631"
format_stamp: "Formatted at 2026-08-06 02:12:03 on dist-test-slave-gp6n"
server_key: "29c636272eaceaaf601bf168f958db49"
server_key_iv: "5e5a8962a05ad6a64346e8d2ada3069b"
server_key_version: "encryptionkey@0"
I20260806 02:12:03.613014 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "5834bfb4e5a7429da3080515dd762631"
format_stamp: "Formatted at 2026-08-06 02:12:03 on dist-test-slave-gp6n"
server_key: "29c636272eaceaaf601bf168f958db49"
server_key_iv: "5e5a8962a05ad6a64346e8d2ada3069b"
server_key_version: "encryptionkey@0"
I20260806 02:12:03.616721 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:12:03.619114 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:03.620303 19675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:12:03.620448 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "5834bfb4e5a7429da3080515dd762631"
format_stamp: "Formatted at 2026-08-06 02:12:03 on dist-test-slave-gp6n"
server_key: "29c636272eaceaaf601bf168f958db49"
server_key_iv: "5e5a8962a05ad6a64346e8d2ada3069b"
server_key_version: "encryptionkey@0"
I20260806 02:12:03.620558 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:03.645190 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:03.648187 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:03.648404 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:03.655901 19675 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:34017
I20260806 02:12:03.655922 19747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:34017 every 8 connection(s)
I20260806 02:12:03.656847 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:03.659525 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:03.660959 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19675
I20260806 02:12:03.661104 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:03.661412 28756 external_mini_cluster.cc:1428] Setting key 03ec1c0d0486c0854a31db42d372f163
I20260806 02:12:03.666098 19748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: Bootstrap starting.
I20260806 02:12:03.668660 19748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:03.669550 19748 log.cc:824] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:03.671643 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:03.662842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58963 (local address 127.28.21.62:34017)
0806 02:12:03.663253 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:03.663271 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:03.663315 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:12:03.664153 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:03.664179 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:03.664597 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:03.664961 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:03.665373 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:03.666814 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:03.667811 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:03.668479 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:03.668678 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:03.669590 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:03.669596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:03.669612 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:03.670095 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:03.670148 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:03.670170 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:03.670245 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:03.670510 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:12:03.670721 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":104,"threads_started":1}
I20260806 02:12:03.671909 19748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: No bootstrap required, opened a new log
I20260806 02:12:03.674482 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:03.675907 19748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } }
I20260806 02:12:03.676118 19748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:12:03.676175 19748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5834bfb4e5a7429da3080515dd762631, State: Initialized, Role: FOLLOWER
I20260806 02:12:03.676692 19748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } }
I20260806 02:12:03.676826 19748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:12:03.676949 19748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:12:03.677063 19748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:03.677973 19748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } }
I20260806 02:12:03.678303 19748 leader_election.cc:302] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5834bfb4e5a7429da3080515dd762631; no voters:
I20260806 02:12:03.678622 19748 leader_election.cc:288] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:12:03.678740 19758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:03.678959 19758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [term 1 LEADER]: Becoming Leader. State: Replica: 5834bfb4e5a7429da3080515dd762631, State: Running, Role: LEADER
I20260806 02:12:03.679343 19758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } }
I20260806 02:12:03.679813 19748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:03.681061 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5834bfb4e5a7429da3080515dd762631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } } }
I20260806 02:12:03.681193 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:03.676527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55327 (local address 127.28.21.62:34017)
0806 02:12:03.676648 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:03.676652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:03.676880 (+ 228us) server_negotiation.cc:407] Connection header received
0806 02:12:03.677029 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:03.677032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:03.677080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:03.677163 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:03.677415 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:03.677923 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:03.679019 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:03.679352 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:03.679436 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:03.680439 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:03.680443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:03.680446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:03.680458 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:03.680486 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:03.680500 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:03.680581 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:03.680959 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:12:03.681012 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:12:03.681360 19760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5834bfb4e5a7429da3080515dd762631. Latest consensus state: current_term: 1 leader_uuid: "5834bfb4e5a7429da3080515dd762631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834bfb4e5a7429da3080515dd762631" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 34017 } } }
I20260806 02:12:03.681792 19760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:03.682039 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631 [sys.catalog]: This master's current role is: LEADER
W20260806 02:12:03.684996 19773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:03.685075 19773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:03.685168 19774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:03.685854 19774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:12:03.690945 19774 catalog_manager.cc:1347] Generated new cluster ID: 0dc5615fea654f9a90804d23670a52b7
I20260806 02:12:03.691027 19774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:03.709455 19774 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:03.710383 19774 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:03.719942 19774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5834bfb4e5a7429da3080515dd762631: Generated new TSK 0
I20260806 02:12:03.720860 19774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:12:03.731168 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19675
2026-08-06T02:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:12:03 dist-test-slave-gp6n krb5kdc[19781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:12:03 dist-test-slave-gp6n krb5kdc[19781](info): set up 2 sockets
Aug 06 02:12:03 dist-test-slave-gp6n krb5kdc[19781](info): commencing operation
krb5kdc: starting...
W20260806 02:12:04.825932 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:04 dist-test-slave-gp6n krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:04.963833 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36711
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:05.078521 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:05.078868 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:05.078954 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:05.083082 19794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:05.083184 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:05.083225 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:05.083262 19794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:05.083292 19794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:05.088852 19794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:36711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:05.090065 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:05.091153 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:05.100186 19800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:05.100174 19801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:05.100171 19803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:05.100345 19794 server_base.cc:1034] running on GCE node
I20260806 02:12:05.100979 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:05.102064 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:05.103236 19794 hybrid_clock.cc:648] HybridClock initialized: now 1785982325103208 us; error 38 us; skew 500 ppm
Aug 06 02:12:05 dist-test-slave-gp6n krb5kdc[19781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:05.111742 19794 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:05.112771 19794 webserver.cc:460] Webserver started at http://127.28.21.62:37873/ using document root <none> and password file <none>
I20260806 02:12:05.113325 19794 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:05.113385 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:05.113559 19794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:05.115890 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "e720b852e5eb4318a75c4ea05ab7b43f"
format_stamp: "Formatted at 2026-08-06 02:12:05 on dist-test-slave-gp6n"
server_key: "2dc36e0aa6a8c66081366ecc65c620c5"
server_key_iv: "95926b45b2d93644b78d42c0f1a6185b"
server_key_version: "encryptionkey@0"
I20260806 02:12:05.116454 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "e720b852e5eb4318a75c4ea05ab7b43f"
format_stamp: "Formatted at 2026-08-06 02:12:05 on dist-test-slave-gp6n"
server_key: "2dc36e0aa6a8c66081366ecc65c620c5"
server_key_iv: "95926b45b2d93644b78d42c0f1a6185b"
server_key_version: "encryptionkey@0"
I20260806 02:12:05.120343 19794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:12:05.122853 19810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:05.124119 19794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:12:05.124264 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "e720b852e5eb4318a75c4ea05ab7b43f"
format_stamp: "Formatted at 2026-08-06 02:12:05 on dist-test-slave-gp6n"
server_key: "2dc36e0aa6a8c66081366ecc65c620c5"
server_key_iv: "95926b45b2d93644b78d42c0f1a6185b"
server_key_version: "encryptionkey@0"
I20260806 02:12:05.124380 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:05.129050 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:05.132304 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:05.132507 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:05.139993 19794 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:36711
I20260806 02:12:05.140022 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:36711 every 8 connection(s)
I20260806 02:12:05.140949 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:05.143778 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:05.149231 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f: Bootstrap starting.
I20260806 02:12:05.150838 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19794
I20260806 02:12:05.151002 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:05.151306 28756 external_mini_cluster.cc:1428] Setting key 07e944208c82ec4aab1c44e64fec0aef
I20260806 02:12:05.151981 19865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:05.152887 19865 log.cc:824] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:05.155055 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f: No bootstrap required, opened a new log
I20260806 02:12:05.156466 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:05.152426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47917 (local address 127.28.21.62:36711)
0806 02:12:05.152846 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:05.152861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:05.152898 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:12:05.153825 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:05.153845 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:05.154114 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:05.154388 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:05.155266 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:05.155271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:05.155282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:05.155292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:05.155345 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:05.155363 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:05.155453 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:05.155605 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:12:05.155773 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260806 02:12:05.159080 19865 raft_consensus.cc:348] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } }
I20260806 02:12:05.159348 19865 raft_consensus.cc:374] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:12:05.159421 19865 raft_consensus.cc:729] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e720b852e5eb4318a75c4ea05ab7b43f, State: Initialized, Role: FOLLOWER
I20260806 02:12:05.159417 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:05.159837 19865 consensus_queue.cc:260] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } }
I20260806 02:12:05.159952 19865 raft_consensus.cc:388] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:12:05.160046 19865 raft_consensus.cc:482] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:12:05.160178 19865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:05.161218 19865 raft_consensus.cc:504] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } }
I20260806 02:12:05.161619 19865 leader_election.cc:302] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e720b852e5eb4318a75c4ea05ab7b43f; no voters:
I20260806 02:12:05.161965 19865 leader_election.cc:288] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:12:05.162024 19876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:05.162231 19876 raft_consensus.cc:686] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [term 1 LEADER]: Becoming Leader. State: Replica: e720b852e5eb4318a75c4ea05ab7b43f, State: Running, Role: LEADER
I20260806 02:12:05.162573 19876 consensus_queue.cc:237] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } }
I20260806 02:12:05.163287 19865 sys_catalog.cc:564] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:05.163511 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:05.161550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49563 (local address 127.28.21.62:36711)
0806 02:12:05.161671 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:05.161675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:05.161886 (+ 211us) server_negotiation.cc:407] Connection header received
0806 02:12:05.162117 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:05.162121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:05.162160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:05.162245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:05.163070 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:05.163073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:05.163076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:12:05.163087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:05.163115 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:12:05.163130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:05.163219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:05.163303 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 02:12:05.163342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:12:05.163756 19876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e720b852e5eb4318a75c4ea05ab7b43f. Latest consensus state: current_term: 1 leader_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } } }
I20260806 02:12:05.163872 19876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:05.164304 19878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e720b852e5eb4318a75c4ea05ab7b43f" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 36711 } } }
I20260806 02:12:05.164418 19878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:05.164778 19880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:05.167927 19880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:12:05.173542 19880 catalog_manager.cc:1347] Generated new cluster ID: 5fd47f72cf8b48dd97677ff4ffa6b9b3
I20260806 02:12:05.173619 19880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:05.181304 19880 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:05.181641 19880 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:05.185539 19880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e720b852e5eb4318a75c4ea05ab7b43f: Generated new TSK 0
I20260806 02:12:05.186189 19880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:12:05.190456 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19794
2026-08-06T02:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1461 ms)
[----------] 18 tests from AuthTokenIssuingTest (26172 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:12:05 dist-test-slave-gp6n krb5kdc[19898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:12:05 dist-test-slave-gp6n krb5kdc[19898](info): set up 2 sockets
Aug 06 02:12:05 dist-test-slave-gp6n krb5kdc[19898](info): commencing operation
krb5kdc: starting...
W20260806 02:12:06.340991 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:06.469648 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45103
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:06.580542 19912 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:06.580865 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:06.580890 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:06.580977 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:06.585210 19912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:06.585290 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:06.585310 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:06.585327 19912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:06.585343 19912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:06.591583 19912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:06.592814 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:06.593910 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:06.603205 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:06.603193 19920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:06.603197 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:06.603531 19912 server_base.cc:1034] running on GCE node
I20260806 02:12:06.604234 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:06.605369 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:06.606560 19912 hybrid_clock.cc:648] HybridClock initialized: now 1785982326606531 us; error 43 us; skew 500 ppm
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:06.609323 19912 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:06.610256 19912 webserver.cc:460] Webserver started at http://127.28.21.62:39075/ using document root <none> and password file <none>
I20260806 02:12:06.610898 19912 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:06.611002 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:06.611240 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:06.613507 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "7d3c32403bb54ded846c5a479bb2b123"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "4c32581acafb3518dcbaff0bbac239ce"
server_key_iv: "795375e5108067e117cff2312d9c7510"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.614120 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "7d3c32403bb54ded846c5a479bb2b123"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "4c32581acafb3518dcbaff0bbac239ce"
server_key_iv: "795375e5108067e117cff2312d9c7510"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.617919 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:12:06.620448 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:06.621693 19912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:12:06.621833 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "7d3c32403bb54ded846c5a479bb2b123"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "4c32581acafb3518dcbaff0bbac239ce"
server_key_iv: "795375e5108067e117cff2312d9c7510"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.621946 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:06.651131 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:06.654121 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:06.654330 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:06.661692 19912 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:45103
I20260806 02:12:06.662024 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:45103 every 8 connection(s)
I20260806 02:12:06.662809 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:06.665666 19982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:06.666869 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19912
I20260806 02:12:06.667052 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:06.667368 28756 external_mini_cluster.cc:1428] Setting key 66187230e0d11f32f690d52190e813e4
I20260806 02:12:06.669888 19982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.61@KRBTEST.COM" created.
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:06.687126 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.672389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45103 (local address 127.0.0.1:54862)
0806 02:12:06.672880 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.672934 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:06.673806 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:06.674684 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:06.674689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:06.675067 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:06.675343 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.675357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.677114 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.677117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.677927 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.677937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.678795 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.678804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.678957 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.679793 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:06.679816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:06.681622 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:06.683913 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.683923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.683934 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.684178 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.684558 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.684561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.684563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.684680 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.685321 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:06.685327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:06.685474 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 02:12:06.685662 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:12:06.686045 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":141,"thread_start_us":133,"threads_started":1}
I20260806 02:12:06.687126 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.672389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.28.21.62:45103)
0806 02:12:06.673025 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:06.673043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:06.673094 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:12:06.674142 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:06.674152 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:06.674430 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:06.674650 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:06.675518 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.676959 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.678068 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.678568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.678843 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.681814 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:06.681820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:06.681831 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:06.681852 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:06.683727 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.684295 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.684301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.684303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.684452 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.684783 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.684787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.684789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.685196 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:06.685341 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:06.685725 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 02:12:06.685990 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":490,"thread_start_us":182,"threads_started":1}
I20260806 02:12:06.689863 19982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:06.690716 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.690342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51366)
0806 02:12:06.690536 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.690648 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260806 02:12:06.691030 19930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
I20260806 02:12:06.692039 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.691809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51382)
0806 02:12:06.691931 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.691976 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:12:06.692255 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:41965: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
I20260806 02:12:06.698035 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:41965
--webserver_interface=127.28.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:12:06.741549 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } attempt: 1
I20260806 02:12:06.742686 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.742274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51386)
0806 02:12:06.742518 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.742597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 02:12:06.743510 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.743264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51394)
0806 02:12:06.743391 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.743448 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:12:06.743652 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:41965: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
I20260806 02:12:06.811816 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } attempt: 2
I20260806 02:12:06.812705 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.812407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51404)
0806 02:12:06.812551 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.812625 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:12:06.813436 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.813199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51416)
0806 02:12:06.813308 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.813382 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:12:06.813570 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:41965: Network error: Client connection negotiation failed: client connection to 127.28.21.61:41965: connect: Connection refused (error 111)
W20260806 02:12:06.818717 19989 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:06.819068 19989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:06.819135 19989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:06.819218 19989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:06.823375 19989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:06.823467 19989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:06.823511 19989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:06.823565 19989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:06.823603 19989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:06.829151 19989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:06.830333 19989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:06.831439 19989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:06.840533 19998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:06.840545 19994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:06.840690 19989 server_base.cc:1034] running on GCE node
W20260806 02:12:06.840574 19995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:06.841410 19989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:06.842641 19989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:06.843844 19989 hybrid_clock.cc:648] HybridClock initialized: now 1785982326843827 us; error 32 us; skew 500 ppm
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:06.846735 19989 init.cc:377] Logged in from keytab as kudu/127.28.21.61@KRBTEST.COM (short username kudu)
I20260806 02:12:06.847711 19989 webserver.cc:460] Webserver started at http://127.28.21.61:44647/ using document root <none> and password file <none>
I20260806 02:12:06.848310 19989 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:06.848407 19989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:06.848650 19989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:06.850845 19989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data/instance:
uuid: "7897cc830f424c5d9cf91e5a41704099"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "6ee809d18480da6b60d7b81f7e1d3832"
server_key_iv: "59e775ccdeaf3175f0098ade7e959d86"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.851436 19989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal/instance:
uuid: "7897cc830f424c5d9cf91e5a41704099"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "6ee809d18480da6b60d7b81f7e1d3832"
server_key_iv: "59e775ccdeaf3175f0098ade7e959d86"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.855254 19989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:12:06.857601 20006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:06.858884 19989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:12:06.859043 19989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "7897cc830f424c5d9cf91e5a41704099"
format_stamp: "Formatted at 2026-08-06 02:12:06 on dist-test-slave-gp6n"
server_key: "6ee809d18480da6b60d7b81f7e1d3832"
server_key_iv: "59e775ccdeaf3175f0098ade7e959d86"
server_key_version: "encryptionkey@0"
I20260806 02:12:06.859161 19989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:06.892391 19989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:06.895833 19989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:06.896044 19989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:06.903422 19989 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.61:41965
I20260806 02:12:06.903740 20061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.61:41965 every 8 connection(s)
I20260806 02:12:06.904500 19989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
I20260806 02:12:06.904721 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } attempt: 3
I20260806 02:12:06.906105 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 19989
I20260806 02:12:06.906256 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-1/wal/instance
I20260806 02:12:06.906670 28756 external_mini_cluster.cc:1428] Setting key 44c223fbaeaaf0414afd923554371218
I20260806 02:12:06.907862 20062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:06.912449 20062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.21.60@KRBTEST.COM; defaulting to no policy
I20260806 02:12:06.921409 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.905330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51430)
0806 02:12:06.905476 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.905505 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:06.905616 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:06.908904 (+ 3288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:06.908911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:06.909147 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:06.909299 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.909306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.911185 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.911191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.912248 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.912267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.913325 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.913332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.913504 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.914162 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:06.914176 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:06.916055 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:06.918789 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.918796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.918799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.919053 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.919491 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.919494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.919496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.919591 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.920598 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:06.920601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:06.920723 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 02:12:06.920946 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 02:12:06.921149 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.60@KRBTEST.COM" created.
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:06.922955 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.906135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.28.21.61:41965)
0806 02:12:06.906669 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:06.906699 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:06.906752 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:12:06.907973 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:06.908000 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:06.908479 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:06.909506 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:06.909854 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.911036 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.912407 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.913160 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.913325 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.916289 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:06.916299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:06.916317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:06.916354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:06.918650 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.919170 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.919177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.919184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.919369 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.919718 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.919722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.919724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.920430 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:06.920578 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:06.921140 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 02:12:06.921783 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":170,"threads_started":1}
I20260806 02:12:06.925490 19982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:06.926323 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.926071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44438)
0806 02:12:06.926179 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.926240 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 02:12:06.927373 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.926906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44444)
0806 02:12:06.927260 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.927304 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
W20260806 02:12:06.927547 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:06.927954 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.914728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45103 (local address 127.0.0.1:54864)
0806 02:12:06.915740 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.915799 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:06.915958 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:06.916352 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:06.916359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:06.916850 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:06.917081 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.917093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.918267 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.918272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.919262 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.919272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.920620 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.920625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.920765 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.921386 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:06.921402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:06.923417 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:06.925765 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.925778 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.925787 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.926122 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.926496 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.926502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.926506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.926602 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.927118 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:06.927127 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:06.927271 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 02:12:06.927434 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 02:12:06.927674 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":166,"threads_started":1}
I20260806 02:12:06.927973 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.915008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.28.21.62:45103)
0806 02:12:06.915146 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:06.915150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:06.915885 (+ 735us) server_negotiation.cc:407] Connection header received
0806 02:12:06.916070 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:06.916075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:06.916128 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:06.916215 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:06.917245 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.918129 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.919378 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.920486 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.920598 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.923580 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:06.923583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:06.923586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:06.923613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:06.925619 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.926244 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.926250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.926255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.926348 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.926710 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.926715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.926718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.926980 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:06.927099 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:06.927730 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 02:12:06.927872 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:12:06.929270 20062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:12:06 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:12:06.939309 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.929773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51432)
0806 02:12:06.929917 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.929933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:06.930018 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:06.930270 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:06.930275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:06.930485 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:06.930627 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.930637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.931412 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.931416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.932335 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:06.932346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.932824 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.932831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:06.932959 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.933539 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:06.933550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:06.935870 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:06.937838 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.937841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.937842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.938035 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.938313 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:06.938315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:06.938317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:06.938474 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:06.938865 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:06.938868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:06.938949 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 02:12:06.939022 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:12:06.939139 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:12:06.939459 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.929864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.28.21.61:41965)
0806 02:12:06.929972 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:06.929982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:06.930000 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:12:06.930115 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:06.930118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:06.930156 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:06.930224 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:06.930748 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.931299 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.932457 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:06.932699 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:06.932781 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:06.936026 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:06.936030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:06.936032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:06.936055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:06.937737 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.938135 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.938138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.938140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.938235 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:06.938550 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:06.938553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:06.938554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:06.938755 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:06.938858 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:06.939069 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 02:12:06.939279 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":42}
I20260806 02:12:06.940389 20062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
I20260806 02:12:06.941241 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.940958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44460)
0806 02:12:06.941072 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.941173 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 02:12:06.941723 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:45299
--webserver_interface=127.28.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:06.941978 20010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:06.943011 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.942743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44464)
0806 02:12:06.942872 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.942934 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:12:06.943166 20062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:06.958714 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 1
I20260806 02:12:06.959671 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.959274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44474)
0806 02:12:06.959433 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.959513 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:12:06.960471 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.960246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44480)
0806 02:12:06.960364 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.960417 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:12:06.960636 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:06.992470 20062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 1
I20260806 02:12:06.993453 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.993087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44490)
0806 02:12:06.993272 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.993347 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:12:06.994184 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:06.993951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44496)
0806 02:12:06.994059 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:06.994115 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:12:06.994318 20062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:07.035799 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 2
I20260806 02:12:07.036777 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.036437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44500)
0806 02:12:07.036638 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.036701 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 02:12:07.037568 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.037371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44510)
0806 02:12:07.037462 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.037519 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:12:07.037762 19982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
W20260806 02:12:07.057117 20070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:07.057430 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:07.057487 20070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:07.057581 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:07.061825 20070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:07.061906 20070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:07.061926 20070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:07.061942 20070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:07.061959 20070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:07.062475 20062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 2
I20260806 02:12:07.063365 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.063012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44518)
0806 02:12:07.063187 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.063258 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:12:07.064126 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.063912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44534)
0806 02:12:07.064012 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.064062 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 02:12:07.064247 20062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:45299: Network error: Client connection negotiation failed: client connection to 127.28.21.60:45299: connect: Connection refused (error 111)
I20260806 02:12:07.068133 20070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:45103,127.28.21.61:41965,127.28.21.60:45299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:07.069334 20070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:07.070557 20070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:12:07.079407 20070 server_base.cc:1034] running on GCE node
W20260806 02:12:07.079348 20075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:07.079355 20076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:07.079357 20078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:07.080238 20070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:07.081476 20070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:07.082897 20070 hybrid_clock.cc:648] HybridClock initialized: now 1785982327082868 us; error 54 us; skew 500 ppm
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:07.085881 20070 init.cc:377] Logged in from keytab as kudu/127.28.21.60@KRBTEST.COM (short username kudu)
I20260806 02:12:07.086958 20070 webserver.cc:460] Webserver started at http://127.28.21.60:41985/ using document root <none> and password file <none>
I20260806 02:12:07.087563 20070 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:07.087657 20070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:07.087913 20070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:07.090127 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data/instance:
uuid: "476ce87803ae41ba966685b128b254a8"
format_stamp: "Formatted at 2026-08-06 02:12:07 on dist-test-slave-gp6n"
server_key: "6bb62aef0dcf4770e155f594eccc8a7f"
server_key_iv: "443435749356dcad0318213dbdf2581a"
server_key_version: "encryptionkey@0"
I20260806 02:12:07.090794 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal/instance:
uuid: "476ce87803ae41ba966685b128b254a8"
format_stamp: "Formatted at 2026-08-06 02:12:07 on dist-test-slave-gp6n"
server_key: "6bb62aef0dcf4770e155f594eccc8a7f"
server_key_iv: "443435749356dcad0318213dbdf2581a"
server_key_version: "encryptionkey@0"
I20260806 02:12:07.094642 20070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:12:07.097123 20085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:07.098348 20070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:12:07.098536 20070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "476ce87803ae41ba966685b128b254a8"
format_stamp: "Formatted at 2026-08-06 02:12:07 on dist-test-slave-gp6n"
server_key: "6bb62aef0dcf4770e155f594eccc8a7f"
server_key_iv: "443435749356dcad0318213dbdf2581a"
server_key_version: "encryptionkey@0"
I20260806 02:12:07.098685 20070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:07.124850 20070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:07.128017 20070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:07.128230 20070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:07.135572 20070 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.60:45299
I20260806 02:12:07.135883 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.60:45299 every 8 connection(s)
I20260806 02:12:07.136610 20070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
I20260806 02:12:07.136919 19982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 3
I20260806 02:12:07.139385 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 20070
I20260806 02:12:07.139515 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785982043785074-28756-0/minicluster-data/master-2/wal/instance
I20260806 02:12:07.139761 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:07.139829 28756 external_mini_cluster.cc:1428] Setting key 419c00c527e56d5acb7fdfbec6e6a055
I20260806 02:12:07.144357 20142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:12:07.150625 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.137461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44550)
0806 02:12:07.137653 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.137670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:07.137783 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:07.140080 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:07.140083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:07.140315 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:07.140447 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.140455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.141937 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.141942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.142654 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.142667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.143887 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.143890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.144002 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.144502 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:07.144521 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:07.146006 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:07.148597 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.148602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.148604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.148817 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.149186 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.149189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.149191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.149333 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.150043 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:07.150045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:07.150129 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:12:07.150240 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:12:07.150398 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 02:12:07.151686 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.138202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.28.21.60:45299)
0806 02:12:07.138624 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.138651 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.138686 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:12:07.139317 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.139333 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.139736 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.140117 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.140575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.141802 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.142800 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.143523 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.143675 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.146232 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.146240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.146260 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.146295 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.148464 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.148929 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.148935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.148941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.149088 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.149443 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.149448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.149450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.149933 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.150064 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.150394 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.150720 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":155,"threads_started":1}
I20260806 02:12:07.153590 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.140678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.28.21.60:45299)
0806 02:12:07.142682 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.142691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.142706 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:12:07.142745 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.142751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.142805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.142899 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.143621 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.144131 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.145371 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.145683 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.145780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.148945 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.148948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.148951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.148973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.151516 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.152035 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.152038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.152040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.152140 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.152471 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.152474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.152476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.152725 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.152896 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.153243 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.153364 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1926,"spinlock_wait_cycles":2176,"thread_start_us":116,"threads_started":1}
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:07.155401 20062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } attempt: 3
I20260806 02:12:07.156770 19982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: Bootstrap starting.
I20260806 02:12:07.158926 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.146959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.28.21.62:45103)
0806 02:12:07.147087 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.147092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.147542 (+ 450us) server_negotiation.cc:407] Connection header received
0806 02:12:07.147682 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.147687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.147739 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.147845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.149246 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.149826 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.151490 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.151828 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.151932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.155186 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.155189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.155192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.155215 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.157045 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.157505 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.157508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.157510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.157595 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.157887 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.157890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.157892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.158146 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.158250 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.158642 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.158765 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:12:07.159711 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.146414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:45103 (local address 127.0.0.1:54874)
0806 02:12:07.147377 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.147443 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:07.147579 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:07.148442 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:07.148448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:07.148925 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:07.149077 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.149084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.149934 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.149937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.151329 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.151353 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.152050 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.152053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.152146 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.152668 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:07.152700 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:07.155027 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:07.157130 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.157140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.157146 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.157392 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.157688 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.157692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.157694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.157783 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.158264 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:07.158270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:07.158387 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 02:12:07.159315 (+ 928us) client_negotiation.cc:240] Negotiation successful
0806 02:12:07.159503 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"spinlock_wait_cycles":896,"thread_start_us":126,"threads_started":1}
I20260806 02:12:07.159785 19982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:07.161073 19982 log.cc:824] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:12:07.161469 20142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:07.163442 19982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: No bootstrap required, opened a new log
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:07.166630 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.155822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44566)
0806 02:12:07.156203 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.156215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:07.156350 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:07.156632 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:07.156635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:07.156824 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:07.156956 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.156966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.157969 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.157972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.158806 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.158817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.159489 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.159495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.159604 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.160093 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:07.160108 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:07.162309 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:07.164730 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.164735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.164739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.165012 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.165380 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.165386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.165391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.165496 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.166015 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:07.166021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:07.166133 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 02:12:07.166229 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:12:07.166389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260806 02:12:07.167528 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.155851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.28.21.60:45299)
0806 02:12:07.155952 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.155956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.156310 (+ 354us) server_negotiation.cc:407] Connection header received
0806 02:12:07.156457 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.156462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.156505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.156585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.157085 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.157806 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.158941 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.159279 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.159846 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.162497 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.162500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.162503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.162527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.164577 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.165147 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.165151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.165153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.165253 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.165622 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.165625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.165627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.165897 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.167088 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.167175 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.167316 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:12:07.168468 19982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.168736 19982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3c32403bb54ded846c5a479bb2b123, State: Initialized, Role: FOLLOWER
I20260806 02:12:07.169108 19982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.169675 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.169806 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.170759 19982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:07.171211 20062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: Bootstrap starting.
I20260806 02:12:07.171988 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.161923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:41965 (local address 127.0.0.1:51440)
0806 02:12:07.162131 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.162145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:07.162217 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:07.162711 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:07.162714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:07.163252 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:07.163360 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.163367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.164389 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.164392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.165238 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.165248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.165910 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.165914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.166017 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.166513 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:07.166526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:07.167936 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:07.170135 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.170138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.170140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.170375 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.170701 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.170704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.170705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.170802 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.171440 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:07.171443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:07.171552 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:12:07.171664 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:12:07.171823 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260806 02:12:07.172555 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.162010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.28.21.61:41965)
0806 02:12:07.162375 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.162379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.162391 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:12:07.162451 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.162454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.162490 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.162558 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.163496 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.164236 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.165376 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.165698 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.165796 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.168302 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.168305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.168308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.168332 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.169961 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.170510 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.170513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.170514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.170597 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.170954 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.170957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.170959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.171236 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.171404 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.172237 (+ 833us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.172357 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260806 02:12:07.173552 20142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:07.174875 20062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: Neither blocks nor log segments found. Creating new log.
W20260806 02:12:07.176045 20165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:07.176116 20165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:07.176564 20062 log.cc:824] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:07.176193 19912 master_runner.cc:186] Error getting master registration for 127.28.21.61:41965: OK, instance_id { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" instance_seqno: 1785982326901349 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:12:07.179765 19912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:12:07.180488 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.158616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.28.21.62:45103)
0806 02:12:07.159069 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.159083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.159094 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:12:07.159125 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.159128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.159167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.159247 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.160243 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.161076 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.162264 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.162681 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.162787 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.166796 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.166801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.166805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.166838 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.177915 (+ 11077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.178568 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.178572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.178574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.178699 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.179045 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.179048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.179050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.179275 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.179449 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.180157 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.180292 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":69,"threads_started":1}
I20260806 02:12:07.182657 20062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: No bootstrap required, opened a new log
I20260806 02:12:07.183931 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.174016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:45299 (local address 127.0.0.1:44580)
0806 02:12:07.174215 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:07.174228 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:07.174309 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:07.175198 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:07.175201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:07.175363 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:07.175448 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.175455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.176445 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.176448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.177260 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:07.177270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.177949 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.177953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:07.178037 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.178524 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:07.178539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:07.180172 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:07.182242 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.182246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.182248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.182510 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.182870 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:07.182873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:07.182875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:07.182964 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:07.183424 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:07.183429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:07.183526 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:12:07.183628 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:12:07.183751 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 02:12:07.184486 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.174866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.28.21.60:45299)
0806 02:12:07.174973 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.174977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.174991 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:12:07.175025 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.175028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.175073 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.175166 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.175573 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.176270 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.177453 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.177758 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.177849 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.180276 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.180280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.180282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.180303 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.182118 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.182619 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.182625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.182630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.182776 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.183066 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.183069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.183071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.183287 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.183387 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.184250 (+ 863us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.184357 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:12:07.187049 20062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.187290 20062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897cc830f424c5d9cf91e5a41704099, State: Initialized, Role: FOLLOWER
I20260806 02:12:07.187642 20062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.188017 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: Bootstrap starting.
Aug 06 02:12:07 dist-test-slave-gp6n krb5kdc[19898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:12:07.189085 20062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:07.191123 20142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:07.192209 20142 log.cc:824] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:07.193293 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.181937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.28.21.61:41965)
0806 02:12:07.182073 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.182078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.182655 (+ 577us) server_negotiation.cc:407] Connection header received
0806 02:12:07.182851 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.182856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.182906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.182962 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.183865 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.184613 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.185659 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.185982 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.186059 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.188990 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.188993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.188996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.189021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.190969 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.191592 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.191597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.191599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.191696 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.192082 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.192086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.192087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.192320 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.192481 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.192964 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.193101 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:12:07.194502 20179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:07.194582 20179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:07.198474 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:07.198690 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.198828 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.199766 19956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7d3c32403bb54ded846c5a479bb2b123"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886
I20260806 02:12:07.200091 19956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:12:07.200224 19956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:07.196240 19989 master_runner.cc:186] Error getting master registration for 127.28.21.60:45299: OK, instance_id { permanent_uuid: "476ce87803ae41ba966685b128b254a8" instance_seqno: 1785982327133531 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:12:07.201345 19989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:12:07.201274 19956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.202143 19956 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [CANDIDATE]: Term 1 election: Requested vote from peers 7897cc830f424c5d9cf91e5a41704099 (127.28.21.61:41965), 476ce87803ae41ba966685b128b254a8 (127.28.21.60:45299)
I20260806 02:12:07.202584 20035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d3c32403bb54ded846c5a479bb2b123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7897cc830f424c5d9cf91e5a41704099"
I20260806 02:12:07.202793 20035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:07.204130 20035 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3c32403bb54ded846c5a479bb2b123 in term 1.
I20260806 02:12:07.204726 19930 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7897cc830f424c5d9cf91e5a41704099, 7d3c32403bb54ded846c5a479bb2b123; no voters:
I20260806 02:12:07.205154 20151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:07.205252 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: No bootstrap required, opened a new log
I20260806 02:12:07.205384 20151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3c32403bb54ded846c5a479bb2b123, State: Running, Role: LEADER
I20260806 02:12:07.205722 20151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.207577 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3c32403bb54ded846c5a479bb2b123. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.207685 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:07.211071 20142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.211341 20142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476ce87803ae41ba966685b128b254a8, State: Initialized, Role: FOLLOWER
I20260806 02:12:07.211695 20142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } }
I20260806 02:12:07.212039 20184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:07.212311 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.212581 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.212993 20184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:12:07.213095 20142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:07.213137 20115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d3c32403bb54ded846c5a479bb2b123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "476ce87803ae41ba966685b128b254a8"
I20260806 02:12:07.213352 20115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:07.214919 20115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3c32403bb54ded846c5a479bb2b123 in term 1.
I20260806 02:12:07.223117 20115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [term 1 FOLLOWER]: Refusing update from remote peer 7d3c32403bb54ded846c5a479bb2b123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:12:07.223825 20182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:12:07.224025 20201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:07.224089 20201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:07.225301 20035 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [term 1 FOLLOWER]: Refusing update from remote peer 7d3c32403bb54ded846c5a479bb2b123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:12:07.225921 20182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:12:07.229229 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3c32403bb54ded846c5a479bb2b123. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.229367 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.229594 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3c32403bb54ded846c5a479bb2b123. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.229751 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.232528 20182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.232650 20182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:07.233727 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.233767 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.233836 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:07.233870 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.234772 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3c32403bb54ded846c5a479bb2b123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3c32403bb54ded846c5a479bb2b123" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 45103 } } peers { permanent_uuid: "7897cc830f424c5d9cf91e5a41704099" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 41965 } } peers { permanent_uuid: "476ce87803ae41ba966685b128b254a8" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 45299 } } }
I20260806 02:12:07.234892 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:07.235265 20210 mvcc.cc:205] Tried to move back new op lower bound from 7315383612298346496 to 7315383612238323712. Current Snapshot: MvccSnapshot[applied={T|T < 7315383612298346496}]
I20260806 02:12:07.235316 20208 mvcc.cc:205] Tried to move back new op lower bound from 7315383612298346496 to 7315383612238323712. Current Snapshot: MvccSnapshot[applied={T|T < 7315383612298346496}]
I20260806 02:12:07.235549 20202 mvcc.cc:205] Tried to move back new op lower bound from 7315383612298346496 to 7315383612238323712. Current Snapshot: MvccSnapshot[applied={T|T < 7315383612298346496}]
I20260806 02:12:07.245076 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.219509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.28.21.60:45299)
0806 02:12:07.219627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.219631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.230714 (+ 11083us) server_negotiation.cc:407] Connection header received
0806 02:12:07.231155 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.231159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.231214 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.231266 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.234058 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.235013 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.238870 (+ 3857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.239203 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.239273 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.241382 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.241386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.241389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.241414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.243093 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.243603 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.243606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.243609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.243699 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.244031 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.244034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.244036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.244250 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.244355 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.244744 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.244875 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:12:07.251993 20184 catalog_manager.cc:1347] Generated new cluster ID: 83fa70e1b88e4d93a96020089d10ffe0
I20260806 02:12:07.252074 20184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:07.254256 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.220378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.28.21.61:41965)
0806 02:12:07.220507 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.220513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.233946 (+ 13433us) server_negotiation.cc:407] Connection header received
0806 02:12:07.234013 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.234015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.234067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.234147 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.236833 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.237560 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.238706 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.239040 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.239120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.250699 (+ 11579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.250703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.250706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.250736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.252432 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.252951 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.252954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.252956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.253064 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.253341 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.253344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.253346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.253545 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.253654 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.253966 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.254099 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:12:07.254895 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:07.219959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.28.21.62:45103)
0806 02:12:07.220074 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.220078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.231068 (+ 10990us) server_negotiation.cc:407] Connection header received
0806 02:12:07.232549 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.232552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.232607 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.232657 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.236332 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.237121 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.238245 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:07.238634 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:07.238710 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:07.249265 (+ 10555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:07.249270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:07.249273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:07.249322 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:07.252806 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.253295 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.253299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.253301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.253392 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:07.253690 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:07.253693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:07.253695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:07.254070 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:07.254205 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:07.254618 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:12:07.254745 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:12:07.316071 20184 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:07.317474 20184 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:07.343724 20184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3c32403bb54ded846c5a479bb2b123: Generated new TSK 0
I20260806 02:12:07.344585 20184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:12:07.375885 20146 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.373257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.28.21.60:45299)
0806 02:12:07.373370 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.373376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.374756 (+ 1380us) server_negotiation.cc:407] Connection header received
0806 02:12:07.374971 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.374975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.375029 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.375085 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.375728 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44588: BlockingRecv error: recv got EOF from 127.0.0.1:44588 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260806 02:12:07.377276 20149 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.372913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.28.21.62:45103)
0806 02:12:07.373075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.373080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.376382 (+ 3302us) server_negotiation.cc:407] Connection header received
0806 02:12:07.376543 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.376547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.376594 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.376641 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.377156 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54906: BlockingRecv error: recv got EOF from 127.0.0.1:54906 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:12:07.378739 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.374100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.28.21.61:41965)
0806 02:12:07.374218 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.374224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.377765 (+ 3541us) server_negotiation.cc:407] Connection header received
0806 02:12:07.377929 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.377933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.377982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.378034 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.378622 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51468: BlockingRecv error: recv got EOF from 127.0.0.1:51468 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 02:12:07.385100 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.383449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.28.21.61:41965)
0806 02:12:07.383558 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.383562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.384086 (+ 524us) server_negotiation.cc:407] Connection header received
0806 02:12:07.384304 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.384309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.384403 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.384460 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.384995 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 02:12:07.390519 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.388742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.28.21.61:41965)
0806 02:12:07.388860 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.388864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.389398 (+ 534us) server_negotiation.cc:407] Connection header received
0806 02:12:07.389576 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.389580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.389635 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.389690 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.390276 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51484: BlockingRecv error: recv got EOF from 127.0.0.1:51484 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 02:12:07.396562 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.394963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.28.21.61:41965)
0806 02:12:07.395067 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.395071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.395629 (+ 558us) server_negotiation.cc:407] Connection header received
0806 02:12:07.395819 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.395822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.395874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.395926 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.396454 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51496: BlockingRecv error: recv got EOF from 127.0.0.1:51496 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 02:12:07.404433 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.402833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.28.21.61:41965)
0806 02:12:07.402970 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.402974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.403466 (+ 492us) server_negotiation.cc:407] Connection header received
0806 02:12:07.403688 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.403692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.403744 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.403836 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.404326 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51512: BlockingRecv error: recv got EOF from 127.0.0.1:51512 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:12:07.417138 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.415209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.28.21.61:41965)
0806 02:12:07.415390 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.415394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.415689 (+ 295us) server_negotiation.cc:407] Connection header received
0806 02:12:07.416022 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.416026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.416117 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.416210 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.417023 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51514: BlockingRecv error: recv got EOF from 127.0.0.1:51514 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:12:07.437536 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.436259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.28.21.61:41965)
0806 02:12:07.436410 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.436415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.436583 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:12:07.436700 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.436703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.436755 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.436852 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.437410 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51528: BlockingRecv error: recv got EOF from 127.0.0.1:51528 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 02:12:07.473109 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.471888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.28.21.61:41965)
0806 02:12:07.472077 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.472082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.472176 (+ 94us) server_negotiation.cc:407] Connection header received
0806 02:12:07.472333 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.472336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.472385 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.472462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.472970 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51532: BlockingRecv error: recv got EOF from 127.0.0.1:51532 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 02:12:07.540448 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.539436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.28.21.61:41965)
0806 02:12:07.539608 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.539612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.539629 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:12:07.539773 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.539776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.539824 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.539898 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.540306 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51538: BlockingRecv error: recv got EOF from 127.0.0.1:51538 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 02:12:07.672230 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.670895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.28.21.61:41965)
0806 02:12:07.671059 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.671064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.671238 (+ 174us) server_negotiation.cc:407] Connection header received
0806 02:12:07.671375 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.671378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.671432 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.671548 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.672092 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51544: BlockingRecv error: recv got EOF from 127.0.0.1:51544 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 02:12:07.932337 20064 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:07.930996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.28.21.61:41965)
0806 02:12:07.931152 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:07.931157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:07.931386 (+ 229us) server_negotiation.cc:407] Connection header received
0806 02:12:07.931577 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:07.931580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:07.931630 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:07.931723 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:07.932206 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51546: BlockingRecv error: recv got EOF from 127.0.0.1:51546 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:12:08.196009 20179 catalog_manager.cc:1259] Loaded cluster ID: 83fa70e1b88e4d93a96020089d10ffe0
I20260806 02:12:08.196110 20179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: loading cluster ID for follower catalog manager: success
I20260806 02:12:08.197947 20179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: acquiring CA information for follower catalog manager: success
I20260806 02:12:08.199261 20179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7897cc830f424c5d9cf91e5a41704099: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:12:08.225639 20201 catalog_manager.cc:1259] Loaded cluster ID: 83fa70e1b88e4d93a96020089d10ffe0
I20260806 02:12:08.225711 20201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: loading cluster ID for follower catalog manager: success
I20260806 02:12:08.227479 20201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: acquiring CA information for follower catalog manager: success
I20260806 02:12:08.228647 20201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 476ce87803ae41ba966685b128b254a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:12:08.451447 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:08.446819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.28.21.61:41965)
0806 02:12:08.447154 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:08.447160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:08.447179 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:12:08.447422 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:08.447426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:08.447560 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:08.447696 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:08.448028 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.448638 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.449644 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.449924 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.450062 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:08.450485 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:08.450980 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:08.451136 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:08.451182 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 02:12:08.451245 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260806 02:12:08.458611 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:08.454932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.28.21.60:45299)
0806 02:12:08.455195 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:08.455199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:08.455213 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:12:08.455296 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:08.455298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:08.455433 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:08.455545 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:08.455742 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.456303 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.457110 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.457341 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.457442 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:08.457762 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:08.458215 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:08.458320 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:08.458356 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 02:12:08.458471 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260806 02:12:08.466096 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:08.461639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.28.21.62:45103)
0806 02:12:08.461937 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:08.461942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:08.461956 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:12:08.462063 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:08.462067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:08.462213 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:08.462334 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:08.462796 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.463636 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.464532 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.464762 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.464864 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:08.465264 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:08.465602 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:08.465681 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:08.465838 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:12:08.465898 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":67,"threads_started":1}
I20260806 02:12:08.466548 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:08.461640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.28.21.60:45299)
0806 02:12:08.461776 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:08.461780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:08.462433 (+ 653us) server_negotiation.cc:407] Connection header received
0806 02:12:08.462614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:08.462617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:08.462679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:08.462753 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:08.463004 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.463722 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.465264 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.465559 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.465673 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:08.466049 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:08.466122 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:08.466168 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:08.466341 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:12:08.466427 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:12:08.466548 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:08.461947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.28.21.61:41965)
0806 02:12:08.462085 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:08.462089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:08.462504 (+ 415us) server_negotiation.cc:407] Connection header received
0806 02:12:08.462725 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:08.462731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:08.462795 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:08.462882 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:08.463160 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.463781 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.465245 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:08.465475 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:08.465532 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:08.466049 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:08.466122 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:08.466168 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:08.466341 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:12:08.466421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:12:08.469312 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19912
I20260806 02:12:08.477614 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 19989
I20260806 02:12:08.484802 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 20070
2026-08-06T02:12:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3295 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:12:08 dist-test-slave-gp6n krb5kdc[20302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:12:08 dist-test-slave-gp6n krb5kdc[20302](info): set up 2 sockets
Aug 06 02:12:08 dist-test-slave-gp6n krb5kdc[20302](info): commencing operation
krb5kdc: starting...
W20260806 02:12:09.566148 28756 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/test-user.keytab.
Aug 06 02:12:09 dist-test-slave-gp6n krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
Entry for principal kudu/127.28.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab.
I20260806 02:12:09.692689 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46151
--webserver_interface=127.28.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:12:09.804924 20316 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:09.805243 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:09.805266 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:09.805361 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:09.809536 20316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:09.809597 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:09.809616 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:09.809655 20316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:09.809689 20316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:09.814915 20316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.62.keytab
--principal=kudu/127.28.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.62:46151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:09.816033 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:09.817090 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:09.825592 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:09.825592 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:09.825783 20316 server_base.cc:1034] running on GCE node
W20260806 02:12:09.825613 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:09.826502 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:09.827582 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:09.828771 20316 hybrid_clock.cc:648] HybridClock initialized: now 1785982329828742 us; error 41 us; skew 500 ppm
Aug 06 02:12:09 dist-test-slave-gp6n krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:09.831612 20316 init.cc:377] Logged in from keytab as kudu/127.28.21.62@KRBTEST.COM (short username kudu)
I20260806 02:12:09.832540 20316 webserver.cc:460] Webserver started at http://127.28.21.62:43835/ using document root <none> and password file <none>
I20260806 02:12:09.833103 20316 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:09.833195 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:09.833426 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:09.835520 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data/instance:
uuid: "d56d13a9663e44d19e82f2ca500dcfc3"
format_stamp: "Formatted at 2026-08-06 02:12:09 on dist-test-slave-gp6n"
server_key: "cd4d9a8db6de00372d5643c9ac32ffb5"
server_key_iv: "95290ddee5350fcee477d0bde9c3ee64"
server_key_version: "encryptionkey@0"
I20260806 02:12:09.836084 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance:
uuid: "d56d13a9663e44d19e82f2ca500dcfc3"
format_stamp: "Formatted at 2026-08-06 02:12:09 on dist-test-slave-gp6n"
server_key: "cd4d9a8db6de00372d5643c9ac32ffb5"
server_key_iv: "95290ddee5350fcee477d0bde9c3ee64"
server_key_version: "encryptionkey@0"
I20260806 02:12:09.839685 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:12:09.842097 20331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:09.843289 20316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:12:09.843446 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal
uuid: "d56d13a9663e44d19e82f2ca500dcfc3"
format_stamp: "Formatted at 2026-08-06 02:12:09 on dist-test-slave-gp6n"
server_key: "cd4d9a8db6de00372d5643c9ac32ffb5"
server_key_iv: "95290ddee5350fcee477d0bde9c3ee64"
server_key_version: "encryptionkey@0"
I20260806 02:12:09.843565 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:09.856117 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:09.859385 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:09.859618 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:09.867452 20316 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.62:46151
I20260806 02:12:09.867738 20383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.62:46151 every 8 connection(s)
I20260806 02:12:09.868543 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/data/info.pb
I20260806 02:12:09.869074 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 20316
I20260806 02:12:09.869228 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-0/wal/instance
I20260806 02:12:09.869524 28756 external_mini_cluster.cc:1428] Setting key e767b0a79cf42a1d077c69e38618d59f
I20260806 02:12:09.871809 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:09.877341 20384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.61@KRBTEST.COM" created.
Aug 06 02:12:09 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:09.893464 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.880184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46151 (local address 127.0.0.1:40902)
0806 02:12:09.880760 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:09.880803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:09.881724 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:09.882473 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:09.882478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:09.882905 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:09.883143 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:09.883156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:09.884373 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:09.884377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:09.885115 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:09.885124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:09.885774 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:09.885778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:09.885899 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:09.886563 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:09.886579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:09.888448 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:09.890544 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:09.890553 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:09.890562 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:09.890816 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:09.891170 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:09.891173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:09.891174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:09.891285 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:09.891923 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:09.891928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:09.892099 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 02:12:09.892321 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 02:12:09.892710 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":92,"spinlock_wait_cycles":32128,"thread_start_us":164,"threads_started":1}
I20260806 02:12:09.893474 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.880184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.28.21.62:46151)
0806 02:12:09.880672 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:09.880691 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:09.880910 (+ 219us) server_negotiation.cc:407] Connection header received
0806 02:12:09.882073 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:09.882077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:09.882263 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:09.882473 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:09.883292 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:09.884213 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:09.885237 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:09.885616 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:09.885787 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:09.888625 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:09.888631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:09.888641 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:09.888664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:09.890352 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:09.890941 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:09.890947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:09.890949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:09.891083 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:09.891370 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:09.891372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:09.891374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:09.891816 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:09.891925 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:09.892389 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 02:12:09.892630 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":317,"thread_start_us":131,"threads_started":1}
I20260806 02:12:09.895953 20384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:09.896766 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.896470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35290)
0806 02:12:09.896585 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:09.896684 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
Entry for principal kudu/127.28.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab.
W20260806 02:12:09.897002 20335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
I20260806 02:12:09.897668 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:33145
--webserver_interface=127.28.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:12:09.897866 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.897635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35292)
0806 02:12:09.897744 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:09.897808 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:12:09.898013 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33145: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
I20260806 02:12:09.947268 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } attempt: 1
I20260806 02:12:09.948254 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.947884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35304)
0806 02:12:09.948079 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:09.948153 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:12:09.949038 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:09.948784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35318)
0806 02:12:09.948900 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:09.948956 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:12:09.949182 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33145: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
W20260806 02:12:10.009053 20391 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:10.009389 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:10.009467 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:10.009550 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:12:10.013446 20391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:10.013541 20391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:10.013581 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:10.013617 20391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:10.013648 20391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:10.017364 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } attempt: 2
I20260806 02:12:10.018249 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.017898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35326)
0806 02:12:10.018080 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.018157 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:12:10.018983 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.018759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35342)
0806 02:12:10.018868 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.018920 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:12:10.019132 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.61:33145: Network error: Client connection negotiation failed: client connection to 127.28.21.61:33145: connect: Connection refused (error 111)
I20260806 02:12:10.019490 20391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.61.keytab
--principal=kudu/127.28.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.61:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:10.020699 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:10.021848 20391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:12:10.030350 20391 server_base.cc:1034] running on GCE node
W20260806 02:12:10.030313 20400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:10.030313 20397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:10.030313 20398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:10.031132 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:10.032227 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:10.033416 20391 hybrid_clock.cc:648] HybridClock initialized: now 1785982330033381 us; error 49 us; skew 500 ppm
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:10.036285 20391 init.cc:377] Logged in from keytab as kudu/127.28.21.61@KRBTEST.COM (short username kudu)
I20260806 02:12:10.037189 20391 webserver.cc:460] Webserver started at http://127.28.21.61:36041/ using document root <none> and password file <none>
I20260806 02:12:10.037755 20391 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:10.037847 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:10.038079 20391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:10.040189 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data/instance:
uuid: "af5e7c18ebe44546b25688b129961ba2"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "5518478198d02053b2d4e8e3e55ec685"
server_key_iv: "d26b9aadb79badde26486162aee976d9"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.040771 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal/instance:
uuid: "af5e7c18ebe44546b25688b129961ba2"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "5518478198d02053b2d4e8e3e55ec685"
server_key_iv: "d26b9aadb79badde26486162aee976d9"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.044554 20391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:12:10.046948 20407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:10.048331 20391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:12:10.048471 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal
uuid: "af5e7c18ebe44546b25688b129961ba2"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "5518478198d02053b2d4e8e3e55ec685"
server_key_iv: "d26b9aadb79badde26486162aee976d9"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.048581 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:10.069710 20391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:10.072757 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:10.072978 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:10.079916 20391 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.61:33145
I20260806 02:12:10.080197 20463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.61:33145 every 8 connection(s)
I20260806 02:12:10.080891 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/data/info.pb
I20260806 02:12:10.083557 20464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:10.084744 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 20391
I20260806 02:12:10.084900 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-1/wal/instance
I20260806 02:12:10.085181 28756 external_mini_cluster.cc:1428] Setting key 7f326dabb2fa0a7998fec2c9cf74ecaf
I20260806 02:12:10.087637 20464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.21.60@KRBTEST.COM" created.
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:10.103042 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.089803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.28.21.62:46151)
0806 02:12:10.089971 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.089976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.091016 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 02:12:10.091970 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.091974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.092048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.092152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.093318 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.093858 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.095169 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.095414 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.095526 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.098084 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.098091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.098097 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.098135 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.100574 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.101110 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.101117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.101123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.101230 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.101640 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.101646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.101651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.101916 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.102036 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.102623 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.102784 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:12:10.103724 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.090323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46151 (local address 127.0.0.1:40908)
0806 02:12:10.090855 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.090897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.091743 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.092334 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.092340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.092701 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.093168 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.093181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.093993 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.093996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.095039 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.095048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.095578 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.095582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.095746 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.096300 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.096316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.097897 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.100700 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.100705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.100724 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.101005 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.101333 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.101335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.101337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.101519 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.102038 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.102043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.102293 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.102528 (+ 235us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.102800 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":159,"threads_started":1}
I20260806 02:12:10.105281 20464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:10.110327 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
Entry for principal kudu/127.28.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab.
I20260806 02:12:10.114085 28756 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
/tmp/dist-test-taskRTVuz1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:38133
--webserver_interface=127.28.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.21.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:12:10.121547 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.105959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.28.21.61:33145)
0806 02:12:10.106371 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.106381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.106544 (+ 163us) server_negotiation.cc:407] Connection header received
0806 02:12:10.106587 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.106592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.106846 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.106990 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.107336 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.108360 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.109227 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.109741 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.109852 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.116937 (+ 7085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.116948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.116962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.116981 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.119657 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.120123 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.120126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.120128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.120251 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.120580 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.120583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.120584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.120763 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.120906 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.121240 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.121372 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":1408,"thread_start_us":141,"threads_started":1}
I20260806 02:12:10.121587 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.105812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35346)
0806 02:12:10.105934 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.105950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.106026 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.107010 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.107012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.107143 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.107231 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.107236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.108489 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.108491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.109090 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.109099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.109859 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.109862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.110071 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.115244 (+ 5173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.115265 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.116781 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.119778 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.119785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.119788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.120019 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.120360 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.120365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.120368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.120472 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.120882 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.120888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.121025 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.121154 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.121323 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":65}
I20260806 02:12:10.123345 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.110969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.28.21.61:33145)
0806 02:12:10.111304 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.111310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.111322 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:12:10.111351 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.111353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.111393 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.111466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.111896 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.112485 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.113507 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.113887 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.114000 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.117568 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.117571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.117573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.117593 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.119435 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.119959 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.119968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.119972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.120085 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.120415 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.120420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.120422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.120763 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.120906 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.123064 (+ 2158us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.123193 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":145,"threads_started":1}
I20260806 02:12:10.123413 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.110886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35350)
0806 02:12:10.111031 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.111048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.111130 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.111512 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.111514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.111674 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.111774 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.111779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.112581 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.112583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.113389 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.113399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.114003 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.114005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.114081 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.115305 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.115326 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.117425 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.119555 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.119559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.119562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.119831 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.120220 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.120223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.120225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.120324 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.122708 (+ 2384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.122713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.122850 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.122985 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.123170 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:12:10.123894 20464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:10.124583 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.124345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44498)
0806 02:12:10.124446 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.124527 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 02:12:10.124665 20384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
W20260806 02:12:10.125005 20411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
I20260806 02:12:10.125450 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.125112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44502)
0806 02:12:10.125301 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.125363 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 02:12:10.125892 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.125665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44506)
0806 02:12:10.125771 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.125814 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:12:10.126060 20464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
I20260806 02:12:10.126330 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.126029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44512)
0806 02:12:10.126196 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.126254 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 02:12:10.126495 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
I20260806 02:12:10.157685 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 1
I20260806 02:12:10.158660 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.158288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44514)
0806 02:12:10.158489 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.158563 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:12:10.159582 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.159333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44526)
0806 02:12:10.159456 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.159516 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:12:10.159729 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
I20260806 02:12:10.175335 20464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 1
I20260806 02:12:10.176339 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.176016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44534)
0806 02:12:10.176186 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.176255 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:12:10.177139 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.176909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44546)
0806 02:12:10.177029 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.177083 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:12:10.177271 20464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
W20260806 02:12:10.234453 20472 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:12:10.234788 20472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:12:10.234858 20472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:12:10.234935 20472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:12:10.234897 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 2
I20260806 02:12:10.235773 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.235430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44562)
0806 02:12:10.235609 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.235681 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:12:10.236519 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.236296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44564)
0806 02:12:10.236404 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.236454 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:12:10.236650 20384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
W20260806 02:12:10.239468 20472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:12:10.239523 20472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:12:10.239543 20472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:12:10.239596 20472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:12:10.239652 20472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:12:10.244911 20472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.21.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.21.62:46151,127.28.21.61:33145,127.28.21.60:38133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/krb5kdc/kudu_127.28.21.60.keytab
--principal=kudu/127.28.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.21.60:38133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 01:37:46 UTC on ccycloud.quasar-negaek.root.comops.site
build id 4569534
I20260806 02:12:10.245429 20464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 2
I20260806 02:12:10.246016 20472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:12:10.246205 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.245882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44568)
0806 02:12:10.246025 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.246116 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:12:10.247054 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.246824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44570)
0806 02:12:10.246928 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.247000 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 02:12:10.247253 20464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.21.60:38133: Network error: Client connection negotiation failed: client connection to 127.28.21.60:38133: connect: Connection refused (error 111)
I20260806 02:12:10.247386 20472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:12:10.256592 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:12:10.256705 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:10.257078 20472 server_base.cc:1034] running on GCE node
W20260806 02:12:10.257966 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:12:10.258679 20472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:12:10.259783 20472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:12:10.260949 20472 hybrid_clock.cc:648] HybridClock initialized: now 1785982330260944 us; error 38 us; skew 500 ppm
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:12:10.263851 20472 init.cc:377] Logged in from keytab as kudu/127.28.21.60@KRBTEST.COM (short username kudu)
I20260806 02:12:10.264760 20472 webserver.cc:460] Webserver started at http://127.28.21.60:40095/ using document root <none> and password file <none>
I20260806 02:12:10.265331 20472 fs_manager.cc:359] Metadata directory not provided
I20260806 02:12:10.265393 20472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:12:10.265627 20472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:12:10.267771 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data/instance:
uuid: "4a64a73365064efda0336695ff1a20f7"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "7d99f29c3a842a832be961844383e345"
server_key_iv: "7c63d95268de514aff6a600aa22a1db2"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.268363 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal/instance:
uuid: "4a64a73365064efda0336695ff1a20f7"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "7d99f29c3a842a832be961844383e345"
server_key_iv: "7c63d95268de514aff6a600aa22a1db2"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.272176 20472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:12:10.274614 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:12:10.275910 20472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:12:10.276160 20472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal
uuid: "4a64a73365064efda0336695ff1a20f7"
format_stamp: "Formatted at 2026-08-06 02:12:10 on dist-test-slave-gp6n"
server_key: "7d99f29c3a842a832be961844383e345"
server_key_iv: "7c63d95268de514aff6a600aa22a1db2"
server_key_version: "encryptionkey@0"
I20260806 02:12:10.276281 20472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:12:10.292284 20472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:12:10.295404 20472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:12:10.295627 20472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:12:10.302734 20472 rpc_server.cc:307] RPC server started. Bound to: 127.28.21.60:38133
I20260806 02:12:10.303066 20541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.21.60:38133 every 8 connection(s)
I20260806 02:12:10.303761 20472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/data/info.pb
I20260806 02:12:10.306717 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:12:10.310704 20542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:12:10.312901 28756 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu as pid 20472
I20260806 02:12:10.313052 28756 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTVuz1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785982043785074-28756-0/minicluster-data/master-2/wal/instance
I20260806 02:12:10.313421 28756 external_mini_cluster.cc:1428] Setting key 57b3d8b610ae00a901c34bae69a9c96f
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:12:10.325419 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.312636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.28.21.62:46151)
0806 02:12:10.312817 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.312823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.313849 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 02:12:10.315189 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.315193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.315255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.315369 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.316207 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.316779 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.317864 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.318118 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.318240 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.321232 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.321235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.321237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.321262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.323017 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.323666 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.323669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.323670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.323747 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.324158 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.324162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.324166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.324408 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.324548 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.325091 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.325241 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:12:10.326287 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.313034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.62:46151 (local address 127.0.0.1:40918)
0806 02:12:10.313654 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.313715 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.314967 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.315427 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.315434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.315801 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.316055 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.316068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.316906 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.316912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.317728 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.317737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.318293 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.318297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.318489 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.319367 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.319384 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.321032 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.323135 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.323147 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.323156 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.323529 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.323851 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.323857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.323861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.324062 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.324553 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.324559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.324804 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.325014 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.325344 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":38,"spinlock_wait_cycles":11648,"thread_start_us":127,"threads_started":1}
I20260806 02:12:10.326287 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.314152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.28.21.60:38133)
0806 02:12:10.314579 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.314588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.314624 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:12:10.314986 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.315006 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.315198 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.315390 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.316240 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.317086 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.317931 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.318301 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.318484 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.321673 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.321681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.321695 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.321723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.323510 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.324017 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.324022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.324023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.324096 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.324392 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.324395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.324397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.324796 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.324944 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.325233 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.325713 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":312,"thread_start_us":138,"threads_started":1}
I20260806 02:12:10.327627 20542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.62@KRBTEST.COM
I20260806 02:12:10.335817 20384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 3
I20260806 02:12:10.337306 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.328440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.28.21.61:33145)
0806 02:12:10.328561 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.328566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.328599 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:12:10.328646 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.328649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.328702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.328787 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.329272 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.330015 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.330919 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.331291 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.331381 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.333514 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.333518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.333520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.333546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.335317 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.335757 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.335761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.335763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.335851 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.336168 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.336172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.336174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.336457 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.336573 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.336956 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.337130 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:12:10.337843 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.328138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.61:33145 (local address 127.0.0.1:35358)
0806 02:12:10.328233 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.328252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.328337 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.328844 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.328848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.329026 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.329137 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.329149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.330120 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.330124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.330791 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.330800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.331418 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.331422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.331540 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.332068 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.332079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.333376 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.335440 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.335444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.335446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.335651 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.335947 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.335951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.335953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.336053 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.336624 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.336628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.336740 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.337540 (+ 800us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.337680 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 02:12:10.337921 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.328580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.28.21.62:46151)
0806 02:12:10.328681 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.328684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.328735 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:12:10.328868 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.328871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.328920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.329006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.329764 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.330381 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.331270 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.331641 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.331774 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.334176 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.334182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.334190 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.334215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.336064 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.336605 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.336609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.336610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.336702 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.337047 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.337050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.337052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.337269 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.337510 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.337631 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.337737 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:12:10.338433 20464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } attempt: 3
I20260806 02:12:10.338692 20542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.62@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.60@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.21.61@KRBTEST.COM for kudu/127.28.21.60@KRBTEST.COM
I20260806 02:12:10.346589 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.336310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.28.21.60:38133)
0806 02:12:10.336419 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.336423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.336498 (+ 75us) server_negotiation.cc:407] Connection header received
0806 02:12:10.336608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.336611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.336655 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.336740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.337173 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.338726 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.339924 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.340393 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.340505 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.343026 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.343029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.343031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.343053 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.344717 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.345248 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.345252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.345253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.345336 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.345649 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.345653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.345655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.345875 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.345968 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.346321 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.346464 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:12:10.346570 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.336267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44594)
0806 02:12:10.336445 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.336457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.336536 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.336756 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.336759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.336928 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.337050 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.337056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.338872 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.338879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.339762 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.339772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.340502 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.340506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.340596 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.341471 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.341484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.342901 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.344875 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.344881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.344882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.345133 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.345451 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.345454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.345456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.345541 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.346021 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.346026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.346130 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.346254 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.346382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
Aug 06 02:12:10 dist-test-slave-gp6n krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785982329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.21.61@KRBTEST.COM
I20260806 02:12:10.350381 20384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: Bootstrap starting.
I20260806 02:12:10.351020 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.339100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44600)
0806 02:12:10.339240 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.339254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.339338 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.340049 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.340052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.340217 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.340275 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.340279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.341968 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.341970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.342827 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.342840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.343466 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.343470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.343825 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.344220 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.344240 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.345671 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.348271 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.348274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.348279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.348541 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.348887 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.348890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.348892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.348992 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.350385 (+ 1393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.350390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.350519 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.350674 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.350832 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:12:10.351076 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.339194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.28.21.60:38133)
0806 02:12:10.339811 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.339830 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.339846 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:12:10.339884 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.339889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.339930 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.340040 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.340388 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.341835 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.342956 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.343341 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.343457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.345792 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.345795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.345797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.345817 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.348161 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.348633 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.348638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.348641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.348745 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.350039 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.350049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.350051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.350221 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.350345 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.350725 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.350899 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":545,"thread_start_us":111,"threads_started":1}
I20260806 02:12:10.353502 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.339374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.21.60:38133 (local address 127.0.0.1:44616)
0806 02:12:10.339523 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 02:12:10.339536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:12:10.339601 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:12:10.340983 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:12:10.340987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:12:10.341194 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:12:10.341307 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.341314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.342681 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.342685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.343594 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:12:10.343604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.344374 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.344380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:12:10.344501 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.344929 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:12:10.344946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:12:10.346911 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:12:10.351219 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.351223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.351225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.351954 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.352346 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:12:10.352350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:12:10.352352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:12:10.352480 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:12:10.352908 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:12:10.352922 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:12:10.353029 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 02:12:10.353146 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:12:10.353317 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:12:10.353580 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.339771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.28.21.60:38133)
0806 02:12:10.340706 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.340719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.340731 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:12:10.340762 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.340768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.340827 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.340927 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.341436 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.342540 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.343743 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.344228 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.344713 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.347335 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.347343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.347348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.347377 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.349926 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.352065 (+ 2139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.352070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.352072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.352210 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.352584 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.352587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.352590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.352791 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.352904 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.353204 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.353410 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":107,"threads_started":1}
I20260806 02:12:10.353682 20384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:10.354135 20542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: Bootstrap starting.
I20260806 02:12:10.354637 20384 log.cc:824] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:10.356827 20384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: No bootstrap required, opened a new log
I20260806 02:12:10.356856 20542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:10.356906 20464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: Bootstrap starting.
I20260806 02:12:10.357213 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.339052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.28.21.61:33145)
0806 02:12:10.339153 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.339158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.339529 (+ 371us) server_negotiation.cc:407] Connection header received
0806 02:12:10.339624 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.339627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.339672 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.339746 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.340653 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.341614 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.342875 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.343224 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.343606 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.349465 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.349469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.349471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.349495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.351828 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.355268 (+ 3440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.355272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.355274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.355380 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.355757 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.355761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.355762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.355999 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.356180 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.356913 (+ 733us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.357069 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:12:10.357672 20542 log.cc:824] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:10.359454 20464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: Neither blocks nor log segments found. Creating new log.
I20260806 02:12:10.359894 20542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: No bootstrap required, opened a new log
I20260806 02:12:10.360257 20464 log.cc:824] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:12:10.360980 20384 raft_consensus.cc:348] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.361227 20384 raft_consensus.cc:729] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56d13a9663e44d19e82f2ca500dcfc3, State: Initialized, Role: FOLLOWER
I20260806 02:12:10.361555 20384 consensus_queue.cc:260] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.362082 20554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.362342 20554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.362480 20464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: No bootstrap required, opened a new log
I20260806 02:12:10.363922 20384 sys_catalog.cc:564] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:10.364660 20542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.364912 20542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a64a73365064efda0336695ff1a20f7, State: Initialized, Role: FOLLOWER
I20260806 02:12:10.365235 20542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.366493 28756 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:12:10.366679 20542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:10.366880 20464 raft_consensus.cc:348] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.367089 20464 raft_consensus.cc:729] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5e7c18ebe44546b25688b129961ba2, State: Initialized, Role: FOLLOWER
I20260806 02:12:10.367425 20464 consensus_queue.cc:260] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.367813 20358 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d56d13a9663e44d19e82f2ca500dcfc3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40934
I20260806 02:12:10.368047 20358 raft_consensus.cc:482] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:12:10.368189 20358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:10.368824 20464 sys_catalog.cc:564] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:12:10.369204 20358 raft_consensus.cc:504] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.370108 20358 leader_election.cc:288] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [CANDIDATE]: Term 1 election: Requested vote from peers af5e7c18ebe44546b25688b129961ba2 (127.28.21.61:33145), 4a64a73365064efda0336695ff1a20f7 (127.28.21.60:38133)
I20260806 02:12:10.370865 20438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5e7c18ebe44546b25688b129961ba2"
I20260806 02:12:10.371093 20438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:12:10.372521 20438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56d13a9663e44d19e82f2ca500dcfc3 in term 1.
I20260806 02:12:10.372999 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4a64a73365064efda0336695ff1a20f7"
I20260806 02:12:10.373239 20515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:12:10.371871 20571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:10.373669 20571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:10.374652 20515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56d13a9663e44d19e82f2ca500dcfc3 in term 1.
I20260806 02:12:10.375177 20335 leader_election.cc:302] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a64a73365064efda0336695ff1a20f7, d56d13a9663e44d19e82f2ca500dcfc3; no voters:
I20260806 02:12:10.374620 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.375540 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.375571 20554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:12:10.375782 20554 raft_consensus.cc:686] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [term 1 LEADER]: Becoming Leader. State: Replica: d56d13a9663e44d19e82f2ca500dcfc3, State: Running, Role: LEADER
I20260806 02:12:10.376080 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.376124 20554 consensus_queue.cc:237] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } }
I20260806 02:12:10.376216 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.378010 20554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56d13a9663e44d19e82f2ca500dcfc3. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.378134 20554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: This master's current role is: LEADER
W20260806 02:12:10.382864 20600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:10.382997 20600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:10.383332 20316 master_runner.cc:263] No leader master found from master d56d13a9663e44d19e82f2ca500dcfc3
I20260806 02:12:10.383412 20316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:12:10.386724 20602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:12:10.387785 20602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 02:12:10.392779 20597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:12:10.392880 20597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:12:10.394268 20438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [term 1 FOLLOWER]: Refusing update from remote peer d56d13a9663e44d19e82f2ca500dcfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:12:10.394268 20515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [term 1 FOLLOWER]: Refusing update from remote peer d56d13a9663e44d19e82f2ca500dcfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:12:10.395004 20599 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:12:10.395242 20554 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:12:10.396452 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.372477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.28.21.61:33145)
0806 02:12:10.372625 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.372630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.383008 (+ 10378us) server_negotiation.cc:407] Connection header received
0806 02:12:10.383226 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.383229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.383297 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.383357 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.384554 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.385431 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.386824 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.387121 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.387239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.389206 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.389210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.389213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.389240 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.391046 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.393737 (+ 2691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.393741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.393744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.393851 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.394198 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.394202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.394204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.394495 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.394609 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.396136 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.396267 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:12:10.398851 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56d13a9663e44d19e82f2ca500dcfc3. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.398976 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.401901 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56d13a9663e44d19e82f2ca500dcfc3. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.402050 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.406894 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.372878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.28.21.62:46151)
0806 02:12:10.373172 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.373178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.390953 (+ 17775us) server_negotiation.cc:407] Connection header received
0806 02:12:10.391714 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.391718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.391774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.391833 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.398494 (+ 6661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.399337 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.400642 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.401011 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.401117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.403127 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.403130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.403133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.403160 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.404915 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.405415 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.405419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.405421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.405514 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.405850 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.405853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.405856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.406084 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.406199 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.406593 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.406717 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260806 02:12:10.410892 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:10.373299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.28.21.60:38133)
0806 02:12:10.373561 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.373567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.390724 (+ 17157us) server_negotiation.cc:407] Connection header received
0806 02:12:10.391979 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.391982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.392043 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.392113 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.397079 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.397988 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.401768 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:10.402178 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:10.402257 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:10.404601 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:12:10.404605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:12:10.404608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:12:10.404636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:12:10.406603 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.407230 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.407234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.407236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.407344 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:12:10.408653 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:12:10.408657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:12:10.408659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:12:10.408898 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:12:10.409044 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:10.410572 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0806 02:12:10.410713 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260806 02:12:10.414006 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.414168 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:10.414318 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.414440 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:12:10.414564 20604 mvcc.cc:205] Tried to move back new op lower bound from 7315383625287716864 to 7315383625224216576. Current Snapshot: MvccSnapshot[applied={T|T < 7315383625287716864}]
I20260806 02:12:10.415016 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.415144 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.416038 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56d13a9663e44d19e82f2ca500dcfc3" member_type: VOTER last_known_addr { host: "127.28.21.62" port: 46151 } } peers { permanent_uuid: "af5e7c18ebe44546b25688b129961ba2" member_type: VOTER last_known_addr { host: "127.28.21.61" port: 33145 } } peers { permanent_uuid: "4a64a73365064efda0336695ff1a20f7" member_type: VOTER last_known_addr { host: "127.28.21.60" port: 38133 } } }
I20260806 02:12:10.416172 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:12:10.416600 20606 mvcc.cc:205] Tried to move back new op lower bound from 7315383625287716864 to 7315383625224216576. Current Snapshot: MvccSnapshot[applied={T|T < 7315383625287716864}]
I20260806 02:12:10.416790 20608 mvcc.cc:205] Tried to move back new op lower bound from 7315383625287716864 to 7315383625224216576. Current Snapshot: MvccSnapshot[applied={T|T < 7315383625287716864}]
I20260806 02:12:10.425742 20602 catalog_manager.cc:1347] Generated new cluster ID: ecd6b80cb91b467cb04fac8a4fdf6276
I20260806 02:12:10.425828 20602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:12:10.444909 20602 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:12:10.446209 20602 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:12:10.459406 20602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d56d13a9663e44d19e82f2ca500dcfc3: Generated new TSK 0
I20260806 02:12:10.460320 20602 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:12:10.495615 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.492693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.28.21.61:33145)
0806 02:12:10.492841 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.492846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.494240 (+ 1394us) server_negotiation.cc:407] Connection header received
0806 02:12:10.494687 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.494691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.494746 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.494816 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.495471 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35376: BlockingRecv error: recv got EOF from 127.0.0.1:35376 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:12:10.496953 20550 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.493491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.28.21.60:38133)
0806 02:12:10.493606 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.493610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.494597 (+ 987us) server_negotiation.cc:407] Connection header received
0806 02:12:10.496161 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.496165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.496220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.496273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.496833 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44646: BlockingRecv error: recv got EOF from 127.0.0.1:44646 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:12:10.499264 20387 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.497910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.28.21.62:46151)
0806 02:12:10.498032 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.498036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.498333 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:12:10.498532 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.498536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.498587 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.498637 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.499147 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40950: BlockingRecv error: recv got EOF from 127.0.0.1:40950 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:12:10.503831 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.502101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.28.21.61:33145)
0806 02:12:10.502208 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.502211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.502780 (+ 569us) server_negotiation.cc:407] Connection header received
0806 02:12:10.503029 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.503033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.503126 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.503181 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.503716 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35378: BlockingRecv error: recv got EOF from 127.0.0.1:35378 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 02:12:10.508971 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.507423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.28.21.61:33145)
0806 02:12:10.507525 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.507529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.508060 (+ 531us) server_negotiation.cc:407] Connection header received
0806 02:12:10.508243 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.508246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.508297 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.508350 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.508867 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35380: BlockingRecv error: recv got EOF from 127.0.0.1:35380 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:12:10.514572 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.513067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.28.21.61:33145)
0806 02:12:10.513162 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.513166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.513671 (+ 505us) server_negotiation.cc:407] Connection header received
0806 02:12:10.513859 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.513862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.513909 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.513958 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.514475 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35390: BlockingRecv error: recv got EOF from 127.0.0.1:35390 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 02:12:10.522122 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.520548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.28.21.61:33145)
0806 02:12:10.520683 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.520687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.521205 (+ 518us) server_negotiation.cc:407] Connection header received
0806 02:12:10.521388 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.521391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.521440 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.521490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.522028 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35400: BlockingRecv error: recv got EOF from 127.0.0.1:35400 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 02:12:10.534287 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.532539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.28.21.61:33145)
0806 02:12:10.532660 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.532667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.533252 (+ 585us) server_negotiation.cc:407] Connection header received
0806 02:12:10.533459 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.533462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.533517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.533572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.534168 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35402: BlockingRecv error: recv got EOF from 127.0.0.1:35402 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:12:10.555343 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.553241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.28.21.61:33145)
0806 02:12:10.553391 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.553396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.554129 (+ 733us) server_negotiation.cc:407] Connection header received
0806 02:12:10.554383 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.554387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.554488 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.554553 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.555213 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35418: BlockingRecv error: recv got EOF from 127.0.0.1:35418 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:12:10.591418 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.590022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.28.21.61:33145)
0806 02:12:10.590156 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.590161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.590445 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:12:10.590634 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.590638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.590696 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.590787 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.591286 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35432: BlockingRecv error: recv got EOF from 127.0.0.1:35432 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 02:12:10.658663 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.657367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.28.21.61:33145)
0806 02:12:10.657539 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.657544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.657697 (+ 153us) server_negotiation.cc:407] Connection header received
0806 02:12:10.657887 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.657890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.657952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.658045 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.658533 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35448: BlockingRecv error: recv got EOF from 127.0.0.1:35448 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:12:10.789975 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:10.788861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.28.21.61:33145)
0806 02:12:10.789036 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:10.789040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:10.789139 (+ 99us) server_negotiation.cc:407] Connection header received
0806 02:12:10.789259 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:10.789262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:10.789321 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:10.789406 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:10.789857 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 02:12:11.050125 20471 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:12:11.048698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.28.21.61:33145)
0806 02:12:11.048865 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.048871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.049058 (+ 187us) server_negotiation.cc:407] Connection header received
0806 02:12:11.049332 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.049335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.049391 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.049472 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:12:11.049951 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35476: BlockingRecv error: recv got EOF from 127.0.0.1:35476 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:12:11.384332 20600 catalog_manager.cc:1259] Loaded cluster ID: ecd6b80cb91b467cb04fac8a4fdf6276
I20260806 02:12:11.384424 20600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: loading cluster ID for follower catalog manager: success
I20260806 02:12:11.386715 20600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: acquiring CA information for follower catalog manager: success
I20260806 02:12:11.387945 20600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a64a73365064efda0336695ff1a20f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:12:11.394119 20597 catalog_manager.cc:1259] Loaded cluster ID: ecd6b80cb91b467cb04fac8a4fdf6276
I20260806 02:12:11.394193 20597 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: loading cluster ID for follower catalog manager: success
I20260806 02:12:11.395998 20597 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: acquiring CA information for follower catalog manager: success
I20260806 02:12:11.397107 20597 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af5e7c18ebe44546b25688b129961ba2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:12:11.568544 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:11.564203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.28.21.61:33145)
0806 02:12:11.564450 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.564455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.564573 (+ 118us) server_negotiation.cc:407] Connection header received
0806 02:12:11.564749 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.564752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.564886 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.565009 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:11.565305 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.565951 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.566991 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.567218 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.567329 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:11.567655 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:11.568114 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:11.568260 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:11.568318 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 02:12:11.568377 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260806 02:12:11.575287 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:11.571615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.28.21.60:38133)
0806 02:12:11.571843 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.571847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.571861 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:12:11.571962 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.571965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.572078 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.572178 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:11.572423 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.573064 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.573955 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.574186 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.574271 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:11.574562 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:11.574971 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:11.575065 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:11.575121 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:12:11.575169 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260806 02:12:11.582439 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:11.578395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.28.21.61:33145)
0806 02:12:11.578533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.578538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.579017 (+ 479us) server_negotiation.cc:407] Connection header received
0806 02:12:11.579142 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.579145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.579185 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.579245 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:11.579541 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.580159 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.581368 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.581599 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.581710 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:11.581960 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:11.582028 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:11.582104 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:11.582201 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:12:11.582251 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 02:12:11.583284 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:11.578617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.28.21.62:46151)
0806 02:12:11.578886 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.578891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.578906 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:12:11.579027 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.579030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.579189 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.579324 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:11.579673 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.580481 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.581531 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.581869 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.582252 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:11.582394 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:11.582751 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:11.582836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:11.582936 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 02:12:11.583009 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260806 02:12:11.583614 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:12:11.578859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.28.21.60:38133)
0806 02:12:11.578961 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 02:12:11.578964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:12:11.579853 (+ 889us) server_negotiation.cc:407] Connection header received
0806 02:12:11.580036 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:12:11.580039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:12:11.580087 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:12:11.580132 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:12:11.580457 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.581209 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.582304 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:12:11.582632 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:12:11.582709 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:12:11.583112 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:12:11.583189 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:12:11.583234 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:12:11.583420 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:12:11.583477 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 02:12:11.586156 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 20316
I20260806 02:12:11.593951 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 20391
I20260806 02:12:11.600636 28756 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTVuz1/build/debug/bin/kudu with pid 20472
2026-08-06T02:12:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3115 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6410 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287754 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST